Learning image data generation device, learning image data generation method, and learning image data generation program

Abstract

To provide a technique that enables efficient additional learning for recognition models that do not have sufficient object detection accuracy during system operation.SOLUTION: An input unit receives an input of a frame image to which metadata containing success / failure information indicating whether or not a recognition model is able to properly detect an object is given. A generation unit generates additional learning image data to which teacher data is given, based on the metadata of the frame image inputted to the input unit, the frame image to which the recognition model is unable to properly detect the object.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technology for constructing a learning image dataset used for additional training of a recognition model that detects objects (people, vehicles, abandoned objects, etc.) captured in frame images of a target area captured by an imaging device. [Background technology]

[0002] Conventionally, there have been recognition models that detect objects captured in frame images of a target area captured by an imaging device (see Patent Documents 1, 2, etc.). Furthermore, as shown in Patent Documents 1, 2, etc., it is known that if the learning images used in machine learning (e.g., deep learning) of the recognition model are not frame images captured in a location where the recognition model detects an object (i.e., if frame images captured in a location other than the location where the recognition model detects an object are used in machine learning), the object recognition accuracy will be low.

[0003] Furthermore, machine learning of a recognition model requires a large amount of training images (for example, training images with training data).

[0004] Patent Documents 1 and 2 are devices that generate training images used in machine learning of a recognition model. The device is configured to generate a composite image as a training image by pasting an image of an object onto a background image of a target area (local background image) captured by an imaging device. Furthermore, training data indicating the type, color, size, and pasting position of the object pasted onto the background image is added to the generated training image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-149086 [Patent Document 2] Japanese Patent Application Publication No. 2022-124739 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in on-site system operation, the object detection accuracy of the recognition model (a recognition model that has undergone machine learning) may not be sufficient. For example, in a system where the target area is outdoors, the frame image of the target area captured by the imaging device is easily affected by the surrounding environment, so the object detection accuracy of the recognition model may not be sufficient.

[0007] Patent Documents 1 and 2 do not have a configuration for performing additional learning on the recognition model to improve the object detection accuracy when it is found that sufficient object detection accuracy is not achieved during system operation.

[0008] An object of the present invention is to provide a technology that can efficiently perform additional learning on a recognition model that does not have sufficient object detection accuracy during system operation. [Means for solving the problem]

[0009] In order to achieve the above object, the learning image data generating device of the present invention is configured as follows.

[0010] This learning data generation device receives frame images of a target area captured by an imaging device and generates a learning image dataset to be used for additional learning of a recognition model that outputs detection results of objects detected through recognition processing performed on the input frame images.

[0011] The input unit receives a frame image with metadata attached, including correct / incorrect information indicating whether the recognition model was able to properly detect an object. That is, the frame image input to the input unit is a frame image on which the recognition model has performed a recognition process to detect an object. The correct / incorrect information is, for example, information indicating correct detection, non-detection, or erroneous detection. Correct detection indicates that the recognition model properly detected the imaged object, non-detection indicates that the recognition model was unable to detect the imaged object, and erroneous detection indicates that the recognition model erroneously detected an object that was not imaged. The metadata also includes the recognition results of the recognition model.

[0012] The generation unit generates additional learning image data to which training data has been added based on metadata of frame images that are input to the input unit and in which the recognition model was unable to properly detect an object.

[0013] The training image dataset referred to here is a collection of one or more training image data. In other words, the training image dataset is a collection of one or more training image data. Training image data is data in which training data indicating information about objects captured in (or combined with) an image (frame image or composite image) is added to the image. The training data is data indicating the position, orientation, type, size, color, etc. of the object.

[0014] In this configuration, during on-site system operation, the generation unit generates additional training image data to be used for additional training of the recognition model based on frame images of a target area where the recognition model was unable to properly detect an object. Therefore, by using the additional training image data generated by the generation unit for additional training of the recognition model, additional training of the recognition model whose object detection accuracy was insufficient can be efficiently performed. In other words, the generation unit can generate an additional training image dataset that can efficiently perform additional training of a recognition model whose object detection accuracy was insufficient.

[0015] The generating unit may also generate learning image data using frame images for which true / false information is undetected or falsely detected as additional learning images.

[0016] The generation unit may also acquire as a background image a frame image captured by an imaging device, where the time difference between the capture time and the frame image for which the correct / incorrect information is undetected is within a predetermined range, and where no object is captured in the image, and generate additional learning image data using as the additional learning image a composite image in which an object of the type that was undetected is composited with the background image acquired here.

[0017] In addition, when the true / false information is a false positive for a frame image and the frame image is a background image in which no object is captured, the generation unit may generate additional learning image data in which the additional learning image is a composite image in which the object is composited with this frame image.

[0018] The system may also include a dataset construction unit that constructs an additional training image dataset by combining the additional training image data generated by the generation unit with training image data used in the previous machine learning of the recognition model. In this case, the dataset construction unit may be configured to construct an additional training image dataset in which the proportion of additional training image data included in the additional training image dataset is adjusted to a predetermined proportion. This configuration prevents additional training of the recognition model from being biased toward frame images in which the recognition model was unable to properly detect objects. [Effects of the Invention]

[0019] According to the present invention, additional learning of a recognition model that does not have sufficient object detection accuracy can be efficiently performed during system operation. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram illustrating an overview of an object detection system. [Figure 2]FIG. 2 is a block diagram showing the configuration of the main parts of the learning image data generation device. [Figure 3] Figure 3(A) is a schematic diagram showing a frame image for which correct / incorrect information was not detected, and Figures 3(B) and (C) are examples of additional learning images created based on the frame image shown in Figure 3(A). [Figure 4] Figure 4(A) is a schematic diagram showing a frame image whose true / false information was a false positive, and Figures 4(B) and (C) are examples of additional learning images generated based on the frame image shown in Figure 4(A). [Figure 5] 10 is a flowchart showing the operation of the learning image data generation device of this example. [Figure 6] 10 is a flowchart showing a first additional learning image data generation process. [Figure 7] 10 is a flowchart showing a second additional learning image data generation process. [Figure 8] FIG. 10 is a block diagram showing the configuration of the main parts of a learning image data generation device according to a first modified example. [Figure 9] 10 is a flowchart showing the operation of the learning image data generation device of Modification 1. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an embodiment of the present invention will be described.

[0022] <1. Application Examples> First, an overview of an object detection system that utilizes a recognition model for detecting an object captured in a frame image of a target area captured by an imaging device will be described.

[0023] 1 is a diagram showing an overview of an object detection system of this example. The object detection system 100 of this example includes an object detection device 1 and a camera 5. The object detection device 1 includes a recognition model 10. The camera 5 corresponds to the imaging device referred to in this invention.

[0024] In this example of object detection system 100, a frame image of a target area captured by camera 5 is input to object detection device 1. Object detection device 1 performs recognition processing to detect objects captured in the input frame image, and outputs the object detection result.

[0025] The camera 5 is installed at an angle that allows the target area to fit within the imaging area. The target area may be, for example, an entrance / exit to a highway, an entrance / exit to a facility, a ticket gate at a station, an entrance to an office building, an entrance to an apartment building, etc. The target area is determined depending on the application of the object detection system 100.

[0026] In this example, camera 5 is a video camera that captures moving images. The frame rate of camera 5 is, for example, several tens of frames / sec (e.g., approximately 10 to 30 frames / sec). Camera 5 outputs frame images of the moving images captured of the target area to object detection device 1.

[0027] The object detection device 1 selects frame images to be processed from among frame images of a target area captured by the camera 5. The object detection device 1 may select the input frame images in order as frame images to be processed, or may select frame images to be processed from the input frame images at intervals of a predetermined number of frames (for example, every 3 frames, every 5 frames), or may select frame images to be processed from the input frame images at predetermined imaging time intervals (for example, every 100 msec, every 300 msec).

[0028] The recognition model 10 included in the object detection device 1 recognizes the position, orientation, type, size, color, etc. of each object captured in a frame image to be processed. The position of the object may be, for example, a position on the frame image, an absolute position in real space, or a relative position with respect to a reference position in real space. Types of objects detected by the recognition model 10 include, for example, vehicles, people, and abandoned objects. The types of objects detected by the recognition model 10 are determined depending on the application of the object detection system 100. The type of objects detected by the recognition model 10 may be one or more.

[0029] The camera 5 may be a digital still camera that captures a still image of the target area when a release signal is input.

[0030] The camera 5 may also be configured to select frame images to be processed from among the frame images of the captured video, and output the selected frame images to be processed to the object detection device 1. In this case, the camera 5 does not output to the object detection device 1 frame images that were not selected as frame images to be processed.

[0031] The object detection device 1 outputs the object detection results obtained by the recognition model 10 to a higher-level device and an alarm device. The higher-level device manages the situation in the target area based on the object detection results obtained by the recognition model 10. The alarm device also estimates whether the situation in the target area is inappropriate based on the object detection results obtained by the recognition model 10, and issues a warning or other notification if it estimates that the situation is inappropriate. An inappropriate situation here refers to, for example, a situation in which a person (pedestrian) walking toward a highway entrance is detected, a situation in which an abandoned object (such as a fallen object) is detected on the road, or a situation in which a suspicious person is detected at the entrance or exit of a facility.

[0032] The training image data generation device in this example generates a training image dataset to be used in machine learning (e.g., deep learning) of the recognition model 10. The training image data generation device generates a training image dataset to be used in the initial machine learning of the recognition model 10 using a method similar to that described in Patent Documents 1 and 2 above. In addition, the training image data generation device generates a training image dataset to be used in the second and subsequent machine learning rounds (corresponding to additional learning as defined in this invention) with the aim of improving the recognition accuracy of the recognition model 10.

[0033] Here, the training image dataset used in the second or subsequent machine learning of the recognition model 10 is referred to as an additional training image dataset. The training image dataset here is a collection of one or more training image data. The training image data is data in which training data is added to an image (frame image or composite image). The training data indicates the position, orientation, type, size, color, etc. of the object captured (or composited) in the image.

[0034] The training image data generation device in this example generates an additional training image data set based on frame images of a target area where the recognition model 10 was unable to properly detect an object during on-site system operation. This allows the training image data generation device to generate an additional training image data set that can efficiently perform additional training to improve the recognition accuracy of the recognition model.

[0035] <2.Configuration example> 2 is a diagram showing the configuration of the main parts of a training image data generation device 20 of this example. The training image data generation device 20 of this example generates a training image dataset to be used for machine learning (e.g., deep learning) of the recognition model 10 included in the object detection device 1 of the object detection system 100 shown in FIG.

[0036] The training image data generating device 20 of this example includes a control unit 21, an input unit 22, an additional training image data set storage unit 23, and an object image database 24 (object image DB 24).

[0037] The control unit 21 controls the operation of each unit in the main body of the learning image data generation device 20. The control unit 21 also has a determination unit 21a and a generation unit 21b. The determination unit 21a and the generation unit 21b included in the control unit 21 will be described later.

[0038] The input unit 22 receives a frame image that has been selected as a frame image to be processed by the recognition model 10 included in the object detection device 1 of the object detection system 100 shown in FIG. 1 and that has undergone recognition processing to detect an object. Metadata including correct / incorrect information is attached to the frame image that is input to the input unit 22. In this example, the correct / incorrect information is information indicating correct detection, non-detection, or erroneous detection. Correct detection indicates a frame image in which the recognition model 10 properly detected an imaged object. Non-detection indicates a frame image in which the recognition model 10 was unable to detect an imaged object. False detection indicates a frame image in which the recognition model 10 erroneously detected an object that was not imaged.

[0039] The metadata also includes, for each object captured in a frame image, the position, orientation, type, size, color, etc. of the object. Furthermore, the metadata also includes the recognition result of the recognition model 10.

[0040] The additional learning image data set storage unit 23 stores additional learning image data used for additional learning of the recognition model 10.

[0041] The object image DB 24 is a database in which images of objects to be pasted onto a background image when generating a composite image are registered.

[0042] Next, the determining unit 21a and generating unit 21b included in the control unit 21 will be described.

[0043] The determination unit 21a determines the classification of a frame image input to the input unit 22. This classification is correct detection, non-detection, or false detection. The determination unit 21a determines the classification of the frame image based on correct / incorrect information included in the metadata of the frame image.

[0044] The generation unit 21b generates additional learning image data to be used for additional learning of the recognition model 10. The generation unit 21b generates additional learning image data to be used for additional learning of the recognition model 10 based on the metadata of a frame image that is input to the input unit 22 and whose true / false information included in the metadata is not a true detection (in other words, a frame image whose true / false information included in the metadata is an undetected or false detection). The additional learning image data is an additional learning image to which training data has been added.

[0045] 3 and 4, the additional learning images generated by the generation unit 21b and used for additional learning of the recognition model 10 will be described. FIG. 3(A) is a schematic diagram showing a frame image for which true / false information was not detected, and FIGS. 3(B) and (C) are examples of additional learning images generated based on the frame image shown in FIG. 3(A). FIG. 4(A) is a schematic diagram showing a frame image for which true / false information was a false positive, and FIGS. 4(B) and (C) are examples of additional learning images generated based on the frame image shown in FIG. 4(A).

[0046] An object 201a captured in a frame image 201 shown in FIG. 3(A) is an object that was not detected (has been undetected) by the recognition model 10. The generation unit 21b sets the capture time of the frame image 201 as a reference time. The generation unit 21b extracts, as a background image, a frame image (a frame image captured by the camera 5) that is input to the input unit 22 and in which no object is captured. For example, the generation unit 21b extracts, as a background image, a frame image that is captured at a time that has the smallest time difference from the reference time, from among the frame images that are input to the input unit 22 and in which no object is captured. The capture time of a frame image to be extracted as a background image may be limited to a time before the reference time or a time after the reference time.

[0047] Furthermore, if the time difference between the capture time of the frame image extracted as the background image and the reference time is not within a predetermined time difference range (for example, within a range of several tens of seconds to several minutes), the generation unit 21b generates a composite image in which the area in which the object is captured in the undetected frame image 201 (the frame image shown in Figure 3(A)) is replaced with the background image of the target area, and sets this composite image as the background image.

[0048] The generation unit 21b generates a composite image in which an image of an object is pasted onto a background image as an image for additional learning for the recognition model 10. The generation unit 21b extracts from the object image DB 24 objects 210a and 220a that are the same type as the object 201a that was not detected by the recognition model 10 and that are similar in size, shape, color, etc.

[0049] The generation unit 21b generates a composite image in which an object 210a is composited with a background image as the additional training image 210 for the recognition model 10, as shown in FIG. 3(B), for example. FIG. 3(B) shows a composite image (additional training image 210) in which objects 210b and 210c are composited with the background image in addition to the object 210a. The generation unit 21b also generates a composite image in which an object 220a is composited with the background image as shown in FIG. 3(C), for example. FIG. 3(C) shows a composite image (additional training image 220) in which object 220b is composited with the background image in addition to the object 220a.

[0050] The objects 210b, 210c, and 220b may be selected according to the object 201a that has not been detected by the recognition model 10, or may be selected randomly regardless of the object 201a.

[0051] 3 does not mean that two frames of additional learning images are generated for a frame image 201 for which true / false information has not been detected, but rather any number of frames of additional learning images may be generated for the frame image 201 as long as they are one or more. The number of frames of additional learning images to be generated for a frame image 201 for which true / false information has not been detected may be determined in advance, or may be determined based on the total number of objects captured in the frame image 201, the number of objects that were not detected, etc.

[0052] Furthermore, frame image 301 shown in FIG. 4(A) is a frame image in which recognition model 10 erroneously detects an object. In this example, recognition model 10 detects an object in area 301a shown in FIG. 4(A). Generation unit 21b determines whether an object is captured in frame image 301 in which recognition model 10 erroneously detects an object. If frame image 301 in which recognition model 10 erroneously detects an object is a frame image in which no object is captured, generation unit 21b generates a composite image in which one or more object images are pasted onto this frame image as an additional training image. Additional training image 310 shown in FIG. 4(B) is a composite image in which object 310a is pasted onto frame image 301 shown in FIG. 4(A), and additional training image 320 shown in FIG. 4(C) is a composite image in which objects 320a and 320b are pasted onto frame image 301 shown in FIG. 4(A).

[0053] Furthermore, if the frame image in which the recognition model 10 erroneously detects an object is a frame image in which the object is captured, the generation unit 21b may use this frame image as an additional learning image, or may generate a composite image in which images of one or more objects are pasted onto this frame image as an additional learning image.

[0054] The objects 310a, 320a, and 320b may be selected in accordance with an object that has been erroneously detected by the recognition model 10, or may be selected randomly regardless of this object.

[0055] 4 does not mean that two frames of additional learning images are generated for a frame image 301 whose true / false information is a false positive, but rather that any number of frames of additional learning images may be generated for the frame image 301 as long as they are one or more. The number of frames of additional learning images to be generated for a frame image 301 whose true / false information is undetected may be determined in advance, or may be determined based on the total number of objects captured in the frame image 301, the number of falsely detected objects, etc.

[0056] The generating unit 21b generates additional training image data by adding training data to the additional training image. The training data indicates the position, type, size, color, etc. of each object in the additional training image.

[0057] The control unit 21 of the training image data generation device 20 is composed of a hardware CPU, memory, and other electronic circuits. When the hardware CPU executes the training image data generation program of the present invention, it operates as a determination unit 21a and a generation unit 21b. The memory also has an area for expanding the training image data generation program of the present invention and an area for temporarily storing data generated during execution of the training image data generation program. The control unit 21 may be an LSI that integrates the hardware CPU, memory, and the like. The hardware CPU is also a computer that executes the training image data generation method of the present invention.

[0058] <3. Example of operation> FIG. 5 is a flowchart showing the operation of the learning image data generating device of this example.

[0059] The training image data generation device 20 deletes all additional training image data stored in the additional training image dataset storage unit 23 (clears the additional training image dataset storage unit 23) (s11). The training image data generation device 20 accepts input of frame images at the input unit 22 (s12). The frame images input to the input unit 22 are frame images that have been selected by the recognition model 10 in the object detection system 100 as frame images to be processed and that have undergone recognition processing to detect objects. In other words, the frame images input to the input unit 22 are frame images captured by the camera 5 during operation of the object detection system 100. In addition, metadata is attached to the frame images input to the input unit 22. This metadata includes, for example, correct / incorrect information (correct detection, non-detection, false detection), the time of image capture, object information related to the captured object, and detection result information related to the detection result of the recognition model 10.

[0060] The learning image data generation device 20 selects a classification determination target image from among the frame images input to the input unit 22 (s13). The determination unit 21a performs the process of s12. In this example, the determination unit 21a selects the frame images as classification determination target images in the order in which they were input to the input unit 22.

[0061] The determination unit 21a determines whether the selected classification determination target image is a correct detection, a non-detection, or a false detection (s14, s15). If the determination unit 21a determines that the frame image is a correct detection, the learning image data generation device 20 determines whether there are any unprocessed frame images (s18).

[0062] When the determining unit 21a determines that the frame image is an undetected frame image, the learning image data generating device 20 performs a first additional learning image data generating process (s16) and proceeds to s 18. The generating unit 21b performs the process of s16.

[0063] 6 is a flowchart showing the first additional learning image data generation process. The generation unit 21b extracts a background image based on metadata assigned to the frame image selected by the determination unit 21a as the current classification determination target image (s21). As described above, the generation unit 21b extracts, as the background image, the frame image whose capture time has the smallest time difference from the reference time (the capture time of the current classification determination target image) from among the frame images input to the input unit 22 and in which no object is captured.

[0064] The generation unit 21b acquires from the object image DB 24 an object image similar to an object that was not detected by the recognition model 10 in the recognition process of the classification determination image (s22). A similar object image is, for example, an object whose appearance (shape, orientation, color, etc.) is similar to that of the currently undetected object. The generation unit 21b pastes the object image acquired in s22 onto the background image extracted in s21 (s23). The position at which the object image acquired in s22 is pasted is the vicinity of the position of the object that was not detected by the recognition model 10. The generation unit 21b also enlarges or reduces the object image acquired in s22 to match the size of the object that was not detected by the recognition model 10. The process of pasting the object image onto the background image by the generation unit 21b is a known technique described in Patent Documents 1 and 2, etc., and therefore will not be described in detail here.

[0065] The generation unit 21b determines whether or not to paste further object images onto the background image extracted in s21 (s24). For example, the generation unit 21b determines the number of objects to be pasted onto the additional learning image to be generated this time based on the number of objects captured in the frame image selected as the classification determination target image this time. If the number of objects to be pasted onto the additional learning image to be generated this time does not reach the number of objects captured in the frame image selected as the classification determination target image this time, the determination unit 21a determines in s24 that further object images will be pasted. The determination criteria in s24 are not limited to the criteria described above.

[0066] When generating unit 21b determines in s24 that an object image should be pasted onto the background image, it acquires an object image from object image DB 24 (s25), pastes the acquired object image onto the background image (s26), and returns to s24. In s25, for example, it may acquire an image of an object of the same type (e.g., a person) as the object that was not detected this time, or it may acquire an image of an object of a different type (e.g., a vehicle).

[0067] The generation unit 21b generates an additional training image by pasting the object image onto the background image. The generation unit 21b generates additional training image data by adding training data related to the object on the additional training image (s27). The generation unit 21b stores the additional training image data generated in s27 in the additional training image dataset storage unit 23 (s28), and ends this process.

[0068] In this way, the generating unit 21b generates, as an additional learning image, a composite image in which an object is pasted onto a background image captured when the surrounding environment is similar to the frame image in which the object went undetected.

[0069] Furthermore, the flowchart shown in FIG. 6 illustrates a process for generating one frame of additional learning image for a frame image in which an object is not detected, but the process from s22 to s28 may be repeated to generate multiple frames of additional learning image for a frame image in which an object is not detected.

[0070] Furthermore, if the determining unit 21a determines that the detection is erroneous, the learning image data generating device 20 performs a second additional learning image data generating process (s17) and proceeds to s 18. The generating unit 21b performs the process of s17.

[0071] 7 is a flowchart showing the second additional learning image data generation process. The generation unit 21b determines whether an object is captured in the classification determination target image based on the metadata assigned to the frame image selected by the determination unit 21a as the classification determination target image this time (s31). If the determination unit 21a determines in s31 that an object is captured, the process proceeds to s34, which will be described later.

[0072] If the determination unit 21a determines in s31 that an object has not been captured, it acquires an object image from the object image DB 24 (s32), pastes the acquired object image on the classification determination target image (s33), and proceeds to s34. In s32, for example, it may acquire an image of an object of the same type (e.g., a person) as the object that was erroneously detected this time, or it may acquire an image of an object of a different type (e.g., a vehicle).

[0073] In s34, generation unit 21b determines whether to further paste an object image onto the classification determination target image. If generation unit 21b determines in s34 that an object image should be further pasted onto the background image, it acquires an object image from object image DB 24 (s35), pastes the acquired object image onto the background image (s36), and returns to s34. In s35, for example, it may acquire an image of an object of the same type (e.g., a person) as the object that was falsely detected this time, or it may acquire an image of an object of a different type (e.g., a vehicle).

[0074] The generation unit 21b assigns training data to the image (additional learning image) to which the object image was pasted in s33 and s36 (s37), stores this as additional learning image data in the additional learning image dataset storage unit 23 (s38), and terminates this process.

[0075] In this way, the generation unit 21b stores, in the additional training image dataset storage unit 23, an image in which an object image has been pasted onto a frame image in which an object has been erroneously detected in the recognition model 10, as an additional training image.

[0076] Furthermore, the flowchart shown in Figure 7 is a process for generating one frame of additional learning image for a frame image in which an object is falsely detected, but it may also be configured to generate multiple frames of additional learning images for a frame image in which an object is falsely detected.

[0077] The training image data generation device 20 determines whether there are any unprocessed frame images in s18. If the training image data generation device 20 determines that there are any unprocessed frame images in s18, it returns to s12 and performs the above-mentioned processing. If the training image data generation device 20 determines that there are no unprocessed frame images in s18, it ends this processing.

[0078] By performing the above processing, the training image data generation device 20 can store additional training image data corresponding to each frame image that is not detected or has been erroneously detected by the recognition model 10 in the additional training image data dataset storage unit 23.

[0079] In this example, the image data for additional learning stored in the additional learning image dataset storage unit 23 is used for additional learning of the recognition model 10. This allows for efficient additional learning of the recognition model 10 whose object detection accuracy is insufficient.

[0080] <4. Modifications> Variation 1 8 is a block diagram showing the configuration of the main parts of a training image data generation device of Modification 1. Training image data generation device 20A of Modification 1 differs from the above example (the example shown in FIG. 2) in that it includes a previous training image dataset storage unit 25. Training image data generation device 20A of Modification 1 also differs from the above example (the example shown in FIG. 2) in that it includes a dataset construction unit 21c in control unit 21.

[0081] In FIG. 8, the same components as those shown in FIG. 2 are denoted by the same reference numerals.

[0082] The previous learning image data set storage unit 25 stores learning image data used in the previous machine learning of the recognition model 10.

[0083] The data set construction unit 21c uses the learning image data stored in the previous learning image data set storage unit 25 to store the learning image data to be used in the current machine learning of the recognition model 10.

[0084] 9 is a flowchart showing the operation of the training image data generation device of Modification 1. As in the above example, the training image data generation device 20A executes the processes related to s11 to s18. If the training image data generation device 20A determines in s18 that there are no unprocessed frame images, it executes an additional training image dataset construction process (s51). The dataset construction unit 21c executes the process related to s51.

[0085] At this point in time, the dataset constructor 21c acquires the number of frames X of the additional learning images stored in the additional learning image dataset storage unit 23. The number of frames X is the number of additional learning image data pieces generated by the generator 21b for the frame images input this time and for which the true / false information was undetected or erroneously detected.

[0086] The data set construction unit 21c calculates the number Y of additional training image data used for additional training of the recognition model 10 using the number of frames X. In this first modification, the data set construction unit 21c calculates the number Y of additional training image data used for additional training of the recognition model 10 as follows: (Number of additional learning image data Y) = (Number of additional learning image data generated this time x Number of frames x) / α That is, in this first modification, an image data set for additional training is generated to be used for additional training of the recognition model 10, in which the proportion of image data for additional training generated for frame images for which the current true / false information was undetected or erroneously detected is α.

[0087] The dataset construction unit 21c extracts (YX) pieces of training image data from the training image data stored in the previous training image dataset storage unit 25, and stores the extracted training image data in the additional training image dataset storage unit 23 as additional training image data.

[0088] When the dataset construction unit 21c completes the processing of s51, it deletes all of the learning image data stored in the previous learning image dataset storage unit 25, and at this point performs an update process to store all of the additional learning image data stored in the additional learning image dataset storage unit 23 (s52), and then terminates this processing.

[0089] The process in s52 may be performed at any time before the process in s11 is performed.

[0090] In this modification 1, additional learning of the recognition model 10 can be performed using an additional learning image data set that includes not only additional learning image data generated for frame images for which the true / false information was undetected or erroneous, but also learning image data used in the previous machine learning. Therefore, in this modification 1, it is possible to prevent the additional learning of the recognition model 10 from becoming biased learning that learns about undetection or erroneous detection. This makes it possible to reduce the frequency of occurrence of a situation in which a frame image that was able to properly detect an object before the additional learning becomes unable to properly detect the object after the additional learning.

[0091] Variation 2 The dataset construction unit 21c may generate an additional training image dataset by combining the additional training image data generated for a frame image for which true / false information was undetected or erroneously detected with all of the training image data stored in the previous training image dataset storage unit 25. Specifically, in s51, the dataset construction unit 21c may store all of the training image data stored in the previous training image dataset storage unit 25 in the additional training image dataset storage unit 23 as additional training image data.

[0092] Even in this second modification, the additional learning of the recognition model 10 can be prevented from becoming biased learning that learns about undetected or erroneous detections.

[0093] It should be noted that this invention is not limited to the above-described embodiments, and that the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. Furthermore, the order of each step in the flowcharts described in all the above examples is merely an example, and may be appropriately changed within the scope of the present invention.

[0094] Furthermore, the correspondence between the configuration according to the present invention and the configuration according to the above-described embodiment can be described as follows: <Additional Notes> <Appendix 1> A learning data generation device (20) that receives a frame image of a target area captured by an imaging device (5) and generates an additional learning image dataset to be used for additional learning of a recognition model (10) that outputs a detection result of an object detected by a recognition process performed on the input frame image, an input unit (22) to which a frame image having metadata including success / failure information indicating whether the recognition model (10) has properly detected an object is input; a generation unit (21b) that generates additional learning image data to which training data is added based on metadata of a frame image in which the recognition model (10) was unable to properly detect an object, which is a frame image input to the input unit (22); A learning image data generation device (20, 20A) comprising:

[0095] <Appendix 2> The correct / incorrect information is a correct detection indicating that the recognition model (10) properly detected the imaged object, a non-detection indicating that the recognition model (10) was unable to detect the imaged object, or a false detection indicating that the recognition model (10) detected an object that was not imaged. The generating unit (21b) generates the additional learning image data for a frame image for which the true / false information is undetected and a frame image for which the true / false information is a false detection. A learning image data generation device (20, 20A) according to appendix 1.

[0096] <Appendix 3> The generating unit (21b) generates learning image data using a frame image for which the true / false information is undetected or erroneously detected as the additional learning image. A learning image data generation device (20, 20A) according to appendix 2.

[0097] <Appendix 4> The generation unit (21b) acquires, as a background image, a frame image captured by the imaging device (5) in which an object is not captured and the time difference between the captured frame image and the frame image in which the true / false information is undetected is smallest, and generates the additional learning image data by using a composite image in which the undetected type of object is composited with the acquired background image as the additional learning image. A learning image data generation device (20, 20A) according to appendix 2 or 3.

[0098] <Appendix 5> When the true / false information is a frame image in which the true / false information is a false detection and the frame image is a background image in which no object is captured, the generation unit (21b) generates the additional learning image data by combining the frame image with the object and using the combined image as the additional learning image. A learning image data generation device (20, 20A) according to any one of Supplementary Notes 2 to 4.

[0099] <Appendix 6> a dataset construction unit (21c) that constructs an additional learning image dataset by combining the additional learning image data generated by the generation unit (21b) with learning image data used in the previous machine learning of the recognition model (10); A learning image data generation device (20, 20A) according to any one of Supplementary Notes 1 to 5.

[0100] <Appendix 7> The dataset construction unit (21c) constructs an additional training image dataset in which a ratio of the additional training image data included in the additional training image dataset is adjusted to a predetermined ratio. A learning image data generation device (20, 20A) according to Supplementary Note 6. [Explanation of symbols]

[0101] 1...Object detection device 5. Camera 10...Recognition model 20, 20A...Learning image data generation device 21...Control unit 21a...judgment section 21b...Generation section 21c…Dataset Construction Department 22...Input section 23... Additional learning image dataset storage unit 24...Object Image Database (Object Image DB) 25…Previous training image dataset storage section 100...Object detection system 201, 301...frame images 210, 220, 310, 320... Additional learning images 201a, 210a, 210b, 220a, 220b…Objects 310a, 320a, 320b…Objects

Claims

1. A learning data generation device that receives input frame images of a target area captured by an imaging device and generates an additional learning image dataset to be used for additional learning of a recognition model that outputs a detection result of an object detected by a recognition process performed on the input frame images, an input unit to which a frame image having metadata attached thereto, the metadata including success / failure information indicating whether the recognition model has properly detected an object, is input; a generation unit that generates additional learning image data by adding training data to additional learning images based on metadata of frame images that are input to the input unit and in which the recognition model was unable to properly detect an object; A learning image data generation device comprising:

2. The correct / incorrect information is a correct detection indicating that the recognition model properly detected the imaged object, a non-detection indicating that the recognition model was unable to detect the imaged object, or a false detection indicating that the recognition model detected an object that was not imaged, the generation unit generates the additional learning image data for a frame image for which the true / false information is undetected and a frame image for which the true / false information is a false detection. The learning image data generating device according to claim 1 .

3. the generation unit generates learning image data using a frame image for which the true / false information is undetected or falsely detected as the additional learning image. The learning image data generating device according to claim 2 .

4. The generation unit acquires, as a background image, a frame image captured by the imaging device, in which an object is not captured and the time difference between the captured frame image and the frame image in which the true / false information is undetected is smallest, and generates the additional learning image data by combining the background image acquired here with the object of the type that was undetected, as the additional learning image. The learning image data generating device according to claim 2 or 3.

5. When the true / false information is a false detection for a frame image that is a background image in which no object is captured, the generation unit generates the additional learning image data using a composite image obtained by combining the frame image with the object as the additional learning image. The learning image data generating device according to claim 2 or 3.

6. A dataset construction unit that constructs an additional training image dataset by combining the additional training image data generated by the generation unit and training image data used in the previous machine learning of the recognition model.

3. The learning image data generating device according to claim 1 or 2.

7. The dataset construction unit constructs an additional training image dataset in which a ratio of the additional training image data included in the additional training image dataset is adjusted to a predetermined ratio. The learning image data generating device according to claim 6.

8. a computer of a learning data generation device that receives frame images of a target area captured by an imaging device and generates an additional learning image dataset to be used in additional learning of a recognition model that outputs a detection result of an object detected by a recognition process performed on the input frame images, A generation process is executed to generate additional learning image data to which training data is added, based on metadata added to a frame image in which the recognition model was unable to properly detect an object, which is input to an input unit. A method for generating image data for learning.

9. A computer of a learning data generation device that receives frame images of a target area captured by an imaging device and generates an additional learning image dataset to be used for additional learning of a recognition model that outputs a detection result of an object detected by a recognition process performed on the input frame images, Execute a generation process to generate additional learning image data to which training data is added, based on metadata added to a frame image in which the recognition model was unable to properly detect an object, which is input to an input unit. A program for generating image data for learning.

Images