Image processing device, image processing system, and image processing method

Abstract

To provide an image processing device using AI technology, which is capable of generating an additional training image independent of experience and know-how, and can improve the accuracy of the additional training image.SOLUTION: An image processing device comprises: an inference unit that, on the basis of an intermediate layer output of an image recognition model obtained by inference using a target image and a candidate image group as input to the image recognition model, derives a feature amount related to the target image and a feature amount related to a candidate image included in the candidate image group, and outputs the feature amounts; and a training image generation unit that generates, in accordance with a feature amount similarity, a training image to be used for learning the target image, the feature amount similarity being a degree of similarity between the feature amount related to the target image and the feature amount related to the candidate image output from the inference unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing system, and an image processing method. [Background technology]

[0002] 2. Description of the Related Art Image processing techniques are known for recognizing the outside world using a camera mounted on a vehicle and using the image processing techniques for autonomous driving and the like. In the image processing technology described above, it is necessary to accurately recognize objects from the obtained images, so AI technology is used and there is a demand for improved re-learning technology for recognizing the outside world.

[0003] In external recognition technology for in-vehicle cameras using AI, the method of collecting and generating additional training images (padded images) to improve images that are recognized incorrectly when evaluating a trained AI model is heavily dependent on the experience and know-how of the developer. For this reason, the accuracy of the additional training images is unstable, and there is a demand for improving the accuracy.

[0004] In other words, a technology is needed that can collect and generate additional learning images without relying on the experience or know-how of the developer, and that can improve accuracy.

[0005] Patent document 1 describes a technology in which a derived image is generated from a training candidate image, the similarity between the training image and the derived image (using the G component of an RGB image) is calculated, the calculated similarity is compared with a judgment threshold, and training candidate images whose similarity is smaller than the judgment threshold are stored as training images.

[0006] Patent document 2 describes a technology that matches a template image region with a candidate image to be added for training, sets a threshold for the matching results, selects all candidate images to be added for training whose similarity with the template image region exceeds the set threshold, and generates training images. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] WO2017 / 109854 publication [Patent Document 2] JP 2022-182149 A Summary of the Invention [Problem to be solved by the invention]

[0008] The similarity between the training image and the derived image described in Patent Document 1 is different from the similarity in AI technology, and it is difficult to apply the technology described in Patent Document 1 to a device that selects training candidate images using AI technology, and there is a possibility that sufficient improvement in accuracy will not be obtained.

[0009] Furthermore, since images are selected based solely on the similarity threshold, the number of selected images may be large, which is not an optimal number of images for re-learning, and it may be difficult to improve accuracy.

[0010] In addition, the similarity described in Patent Document 2 is different from the similarity in AI technology, and like the technology described in Patent Document 1, it is difficult to apply the technology described in Patent Document 2 to a device that selects learning candidate images using AI technology, and there is a possibility that sufficient improvement in accuracy will not be obtained.

[0011] Furthermore, since images are selected based solely on the similarity threshold, the number of selected images may be large, which is not an optimal number of images for re-learning, and it may be difficult to improve accuracy.

[0012] An object of the present invention is to provide an image processing device and an image processing method that use AI technology and are capable of generating additional training images without relying on experience or know-how and improving the accuracy of the additional training images. [Means for solving the problem]

[0013] In order to achieve the above object, the present invention is configured as follows.

[0014] The image processing device includes an inference unit that determines features related to the target image and features related to candidate images included in the group of candidate images based on an intermediate layer output of an image recognition model obtained by inference using a target image and a group of candidate images as inputs to the image recognition model, and outputs the features, and a learning image generation unit that generates a learning image to be used for learning the target image according to feature similarity, which is the degree of similarity between the features related to the target image and the features related to the candidate images output from the inference unit. Effect of the Invention

[0015] According to the present invention, it is possible to provide an image processing device and an image processing method using AI technology that can generate additional training images without relying on experience or know-how and can improve the accuracy of the additional training images. [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a functional configuration of an image processing apparatus according to a first embodiment. [Diagram 2] FIG. 11 is a block diagram showing a functional configuration of an image processing apparatus according to a second embodiment. [Diagram 3] FIG. 13 is a diagram showing the characteristics of the number of images required to be added depending on the similarity with an image to be subjected to accuracy improvement. [Figure 4] FIG. 11 is a block diagram showing a functional configuration of an image processing apparatus according to a third embodiment. [Diagram 5] FIG. 13 is a diagram illustrating a method for determining the number of sheets to be selected according to similarity and reliability. [Figure 6] FIG. 13 is a block diagram showing a functional configuration of an image processing device according to a first modified example. [Figure 7] FIG. 11 is a block diagram showing a functional configuration of an image processing device according to a second modified example. [Figure 8] FIG. 11 is a system configuration diagram showing a second modified example. [Figure 9] 1 is a hardware configuration diagram of an image processing system including an image processing device and a server according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, in the embodiments of the present invention other than the first embodiment, the same parts as those in the first embodiment are designated by the same reference numerals, and detailed description thereof will be omitted. EXAMPLES

[0018] Example 1 FIG. 1 is a block diagram showing the functional configuration of an image processing device 100 and a server 500 according to a first embodiment of the present invention.

[0019] The image processing device 100 shown in FIG. 1 includes an inference unit 1, a learning image generation unit 2, and a storage unit 3. The learning image generation unit 2 includes a similarity calculation unit 21 and a similar image selection unit 22. The storage unit 3 includes a learning candidate image group storage unit 31, a similarity threshold storage unit 32, an accuracy improvement target image storage unit 33, and a learning image storage unit 34. The learning candidate image group storage unit 31, the similarity threshold storage unit 32, the accuracy improvement target image storage unit 33, and the learning image storage unit 34 may be different memory areas of the same storage unit, or may be different storage units. For example, as described later, when the image processing device 100 is an on-board ECU (electronic control unit) mounted on an automobile, the learning candidate image group storage unit 31 and the similarity threshold storage unit 32 can be provided by the on-board ECU, and the accuracy improvement target image storage unit 33 and the learning image storage unit 34 can be provided by a server connected to the on-board ECU for communication. The hardware configuration of the image processing device 100 will be described later.

[0020] 1 includes a learning unit 510 and a storage unit 520. The storage unit further includes a model storage unit 530. The server 500 is connected to an image processing device via a network.

[0021] The inference unit 1 executes a recognition process that recognizes a predetermined object or information from image data using an image recognition model. A single image to be improved that is stored in an image storage unit 31 to be improved and multiple learning candidate images that are stored in a learning candidate image group storage unit 33 are input to the inference unit 1. For example, a classification AI model can be used for the image recognition model 30, but is not limited to this. Note that the number of images to be improved that are input to the inference unit 1 is limited to one, but may be multiple.

[0022] The inference unit 1 receives an image to be improved and a candidate image for learning as input, executes calculation processing using an AI model, and calculates an intermediate layer output. The intermediate layer output is, for example, the final convolution layer of a classification AI model, but is not limited to this, and may be any output value in the intermediate layer of the AI ​​model. This intermediate layer output is hereinafter referred to as a "feature amount." The inference unit 1 inputs the feature amount obtained by calculation using the image to be improved as input and the feature amount obtained by calculation using the candidate image for learning as input to the learning image generation unit 2.

[0023] The learning image generating unit 2 includes a similarity calculating unit 21 and a similar image selecting unit 22. The similarity calculating unit 41 calculates the similarity between a feature corresponding to an image to be subjected to accuracy improvement and a feature corresponding to a learning candidate image (hereinafter referred to as feature similarity). Here, an example of a method for calculating the feature similarity is shown. The method for calculating the feature similarity is not limited to the following example.

[0024] Assume that the feature amounts corresponding to the target image for accuracy improvement calculated by the inference unit 1 are A1, A2, A3,...A64. Assume that the feature amounts of the candidate images for learning calculated by the inference unit 1 are B1, B2, B3,...B64. In this case, the feature similarity is calculated by the following formula (1).

[0025] Feature similarity = (A1-B1)^2+(A2-B2)^2+(A3-B3)^2+ +(A64-B64)^2 (Formula 1) Next, the similar image selection unit 22 refers to a similarity threshold previously stored in the similarity threshold storage unit 33, and selects a learning image from the multiple learning candidate images based on the similarity threshold. The learning image generation unit 2 selects all or some of the multiple learning candidate images whose feature similarity exceeds the similarity threshold as learning images. Here, one similarity threshold stored in the similarity threshold storage unit 33 is used, but multiple similarity thresholds may be used. An embodiment using multiple similarity thresholds will be described in Example 2. Details of the similarity threshold will be described later.

[0026] The storage unit 3 includes a learning candidate image group storage unit 31, an accuracy improvement target image storage unit 32, a similarity threshold storage unit 33, and a learning image storage unit 34. The learning candidate image group storage unit 31 is a storage unit that stores learning candidate images, which are any image group that are candidates for images used in learning an AI model (image recognition model) for an accuracy improvement target image. The learning candidate images may be images automatically generated by a known image generation technology, or may be images captured by a camera. For example, the learning candidate images may be multiple images captured by a camera mounted on a vehicle of a traffic scene or environment while the vehicle is traveling.

[0027] The accuracy improvement target image storage unit 32 is a storage unit that stores an accuracy improvement target image, which is a target image for improving the accuracy of the recognition process by the image recognition model. The accuracy improvement target image stored in the accuracy improvement target image storage unit 32 may be one or more. The accuracy improvement target image may be, for example, an image that has been recognized incorrectly during evaluation of an arbitrary trained image recognition model.

[0028] The similarity threshold storage unit 33 is a storage unit that stores a similarity threshold that serves as a criterion when the learning image generation unit 2 selects a learning image from the learning candidate images. An example of a method for determining the similarity threshold is shown below. Note that the step of determining the similarity threshold stored in the similarity threshold storage unit 33 may be executed in the image processing device 100, may be executed by a different computer having the same functional configuration as the image processing device 100, or may be partially executed in the image processing device 100.

[0029] First, a feature similarity is calculated, which is the degree of similarity between the features corresponding to the image to be improved obtained by the inference unit 1 and the features corresponding to multiple candidate training images, and the candidate training images are sorted in order of increasing feature similarity (i.e., decreasing value obtained from Equation 1) (Step 1).

[0030] Next, an arbitrary similarity threshold is set as a provisional similarity threshold (step 2).

[0031] Based on the set provisional similarity threshold and the feature similarity, one or more learning images are selected from the multiple learning candidate images (step 3).

[0032] The image recognition model is retrained based on the one or more candidate learning images selected here and the trained images used to train the image recognition model, to generate a retrained image recognition model (step 4).

[0033] Finally, the inference unit 1 inputs an evaluation image for evaluating the recognition accuracy of the re-trained image recognition model and an image to be improved in accuracy to the re-trained image recognition model, and executes inference (step 5).

[0034] If the inference unit 1 determines that the recognition accuracy of the image to be improved and the evaluation image exceeds the threshold, the provisional similarity threshold is set as the formal similarity threshold. If the inference unit 1 determines that the recognition accuracy of the image to be improved and the evaluation image falls below the threshold, the process returns to step 2, the provisional similarity threshold is reset, and steps 3 to 5 are executed again. This is repeated until the inference unit 1 determines that the recognition accuracy of the image to be improved and the evaluation image exceeds the threshold (step 5).

[0035] Returning to FIG. 1, the learning image generation unit 2 stores the learning candidate images selected from the learning candidate images in the learning image storage unit 34 as learning images to be used for learning an image recognition model.

[0036] The learning unit 510 reads out the learning images stored in the learning image storage unit 34. Then, the learning unit 510 re-learns the image recognition model stored in the model storage unit 530 with the learned images and the read-out learning images, and stores the re-learned image recognition model in the model storage unit.

[0037] The model storage unit 530 stores an image recognition model used for inference in the inference unit 1. The image processing device 100 stores the re-learned image recognition model re-learned by the learning unit 510 in the memory unit 3, and can use it for inference in the inference unit 1.

[0038] According to the above-described first embodiment, it is possible to generate learning images capable of improving the learning accuracy of an image recognition model without relying on experience or know-how. That is, by generating learning images based on the similarity (feature similarity) of intermediate layer outputs obtained by executing arithmetic processing using an image recognition model, rather than the similarity of image feature values ​​of the images themselves between an accuracy improvement target image and a group of learning candidate images, it is possible to generate learning images capable of improving the learning accuracy of an image recognition model.

[0039] Example 2 Next, a second embodiment of the present invention will be described.

[0040] FIG. 2 is a functional block diagram of an image processing device 101 according to the second embodiment.

[0041] The difference between the first embodiment shown in FIG. 1 and the second embodiment shown in FIG. 2 is that a generation number adjustment unit 4 is added.

[0042] 2, the similarity calculation unit 21 calculates the similarity of the features (feature similarity) between the learning candidate image and the accuracy improvement target image based on the feature output from the inference unit 1. The similarity calculation unit 21 outputs the calculated feature similarity to the similar image selection unit 22 and also outputs it to the generation number adjustment unit 4. The similar image selection unit 22 selects a learning candidate image whose feature similarity exceeds a similarity threshold, and outputs it to the generation number adjustment unit 4.

[0043] The generation number adjustment unit 4 determines the number of images or the amount of data to be selected according to the feature similarity from the learning candidate images output from the learning image generation unit 2. Then, the generation number adjustment unit 4 stores the learning images in the learning image storage unit 34 according to the determined number of images or amount of data.

[0044] FIG. 3 is a diagram showing the characteristics of the number of images required to be added depending on the similarity with the image to be subjected to accuracy improvement, and explaining a method of determining the number of images to be selected depending on the feature amount similarity.

[0045] FIG. 3 shows the relationship between the feature similarity and the number of images required. When the target image for accuracy improvement and the candidate image for learning are 100% similar (i.e., identical), one additional image is sufficient for retraining the image recognition model. When the target image for accuracy improvement and the candidate image for learning are not similar at all (0% similar), theoretically, even if the number of images added is infinite, there is no effect of retraining the image recognition model. Here, the specific value of the number of images required (required data amount) is related to the number of trained images used in training the image recognition model. The larger the number of trained images, the more the relationship between the feature similarity and the number of additional images required moves in parallel upward on the graph. For example, in order to prevent overtraining from occurring for a specific target image for accuracy improvement, the number of training images to be added is about 5% of the number of trained images used in training the image recognition model.

[0046] As shown in FIG. 3, the generation number adjustment unit 4 sets the number of learning images or the data amount of learning images so that the number of images required decreases as the average feature amount similarity (average feature amount similarity) between the feature amount corresponding to the accuracy improvement target image and the multiple feature amounts corresponding to the multiple learning candidate images increases. The number of learning images or the data amount is set, for example, by changing the similarity threshold. When the number of learning images or the data amount for one accuracy improvement target image is to be greater than the other accuracy improvement target images, the similarity threshold for one accuracy improvement target image is set lower than the similarity threshold for the other accuracy improvement target images. That is, a different similarity threshold can be set for each accuracy improvement target image. The method of setting the number of learning images or the data amount is not limited to the above-mentioned method. The generation number adjustment unit 4 can use not only the average value of multiple feature amount similarities but also any index (such as the median) statistically obtained from the feature amount similarities.

[0047] In the second embodiment, in addition to obtaining the same effect as in the first embodiment, more appropriate learning images can be generated. That is, by adjusting the number of learning images or the amount of data by the generation number adjustment unit 4, it is possible to prevent overlearning from occurring when re-learning an image recognition model using the generated learning images. Details of the effect will be described below.

[0048] In the case of the first embodiment, a learning image is selected from the learning candidate images based on one similarity threshold. However, in general, it is assumed that there are a plurality of accuracy improvement target images, and a learning image needs to be selected from the learning candidate images for each accuracy improvement target image. Here, it is assumed that the level (distribution) of the feature similarity between each accuracy improvement target image and the learning candidate image is different. In this case, if a learning image for each accuracy improvement target image is selected based on one similarity threshold, a large number of learning images may be generated for a specific accuracy improvement target image. As a result, the image recognition model after re-learning, which is re-learned by the learning unit 510 using the learning images output from the learning image generation unit 2, can recognize the accuracy improvement target image from which a large number of learning images are extracted with high accuracy. However, due to over-learning caused by the generation of a large number of learning images, the recognition accuracy of the evaluation image that was recognized before re-learning may decrease. In other words, the recognition accuracy of the evaluation image by the image recognition model after re-learning by the learning unit 510 may decrease compared to the recognition accuracy of the evaluation image by the image recognition model before re-learning by the learning unit 510.

[0049] Therefore, in the second embodiment of the present invention, when there are multiple accuracy improvement target images, the average feature similarity, which is the similarity of the feature of the learning image to the feature of each accuracy improvement target image, is calculated, and a restriction is imposed on the number of learning images or the amount of data to be generated based on the relationship shown in Fig. 3. This makes it possible to prevent overlearning in the image recognition model after re-learning. Therefore, it is possible to generate learning images that can improve the learning accuracy of the image recognition model.

[0050] Example 3 Next, a third embodiment of the present invention will be described.

[0051] FIG. 4 is a configuration diagram of an image processing device 102 according to the third embodiment.

[0052] The difference between the second embodiment shown in FIG. 2 and the third embodiment shown in FIG. 4 is that the inference unit 1 uses the result of inference of the target image for accuracy improvement as input as the reliability, and the generation number adjustment unit 4 determines the number of learning images to be generated or the amount of data based on not only the feature similarity but also the reliability. The reliability here refers to the recognition confidence of the image recognition model for the generated recognition result when the target image for accuracy improvement is inferred by the image recognition model. The higher the reliability, the higher the confidence of the image recognition model, meaning that the recognition result is more likely to match the correct answer. It is also called a confidence value. For example, if the reliability is 50% or less, the reliability is low, and if the reliability of the inference result by the inference unit 1 is 50% or less, the image on which the inference was performed (the image input to the inference unit 1) becomes the target image for accuracy improvement.

[0053] The inference unit 1 outputs the feature amount (intermediate layer output) to the learning image generation unit 2, and further outputs the result of inference using the image to be improved as input to the number of images to be generated adjustment unit 4 as the reliability of the image to be improved.

[0054] The generation number adjustment unit 4 determines the number of learning images to be selected as learning images from the learning candidate images output from the learning image generation unit 2, depending on the feature similarity and the reliability.

[0055] FIG. 5 is a diagram showing the characteristics of the number of images required depending on the feature similarity and reliability of the images to be improved, and explaining a method for determining the number of learning images or the amount of data to be selected depending on the feature similarity and reliability.

[0056] 5, the generation number adjustment unit 4 sets the number of learning images or the amount of data so that the higher the feature similarity, the fewer the number of images required, and the higher the reliability, the fewer the number of images required. For example, when the reliability of a specific accuracy improvement target image is half that of other accuracy improvement target images, the generation number adjustment unit 4 sets the required number of images or the required amount of data so that the required number of learning images to be generated for the specific accuracy improvement target image is twice that of the other accuracy improvement target images.

[0057] In the third embodiment, it is possible to generate a more appropriate number of learning images than in the second embodiment. In other words, by re-learning the image recognition model by the learning unit 510 using the learning images generated according to the number of images or the amount of data determined by the generation number adjustment unit 4, it is possible to prevent over-learning from occurring in the image recognition model after re-learning. Details of the effect will be described below.

[0058] As described above, the higher the reliability, the higher the confidence of the AI ​​model, and the higher the possibility that the recognition result and the correct answer value will match. Conversely, the lower the reliability, the lower the confidence of the AI ​​model, and the lower the possibility that the recognition result and the correct answer value will match. Here, the accuracy improvement target image with a low reliability may be an image with features that the image recognition model has never learned or has only a small amount of learning. Therefore, for accuracy improvement target images with low reliability, a large number of learning images with similar features to the accuracy improvement target image are required when retraining the image recognition model. On the other hand, for accuracy improvement target images with high reliability, a large number of learning images similar to the accuracy improvement target image may not be required when retraining the image recognition model. If learning images are given when an image recognition model is used for an accuracy improvement target image with high reliability, overlearning may occur, and the recognition accuracy of the evaluation image by the image recognition model after retraining may decrease compared to the recognition accuracy of the evaluation image by the image recognition model before retraining.

[0059] Therefore, in the third embodiment of the present invention, when there are multiple images to be improved, the reliability of the images to be improved is calculated from the result of inference by the inference unit 3, and a restriction is imposed on the number of learning images or the amount of data to be generated based on the relationship shown in Fig. 5. This makes it possible to prevent overlearning in the re-learned image recognition model generated by re-learning by the learning unit 510.

[0060] As described above, according to the present invention, a learning image to be used for learning a target image is generated according to feature similarity, which is the degree of similarity between features related to a target image and features related to a candidate image. Therefore, it is possible to provide an image processing device, image processing system, and image processing method using AI technology that can generate additional learning images without relying on experience or know-how and improve the accuracy of the additional learning images.

[0061] (Variation 1) Next, a first modified example of the image processing device 100 will be described.

[0062] Fig. 6 is a configuration diagram showing a first modified example of the image processing device 100. As shown in Fig. 6, the learning unit 510 and the model storage unit 530 may be provided in the image processing device 100. This allows the image processing device 100 to execute re-learning of the image recognition model for a desired target image for accuracy improvement without communicating with a server, thereby improving the recognition accuracy of the image recognition model.

[0063] (Variation 2) Next, a second modified example of the image processing system including the image processing device 100 and the server 500 will be described.

[0064] FIG. 7 is a configuration diagram showing a second modified example of an image processing system including an image processing device 100 and a server 500. As shown in FIG. 7, the accuracy improvement target image storage unit 32 and the learning image storage unit 34 may be included in the storage unit 520 of the server 500. Furthermore, the image processing device 100 may be included in the vehicle 900, and may be communicatively connected to a camera 99 mounted on the vehicle 900. In this case, the inference unit 1 can use the captured image captured by the camera 99 as a learning candidate image instead of the learning candidate image stored in the learning candidate image group storage unit 31, or in combination with the learning candidate image stored in the learning candidate image group storage unit 31. The camera 99 may be a single monocular camera, a stereo camera consisting of a pair of left and right cameras, or a multi-camera installed at different positions on the vehicle 900.

[0065] FIG. 8 is a diagram showing an example of a system configuration in which the image processing device 100 is provided in a vehicle 900. As shown in FIG.

[0066] In a vehicle 900, an image processing device 100 is connected to a camera 99, a vehicle control device 90, a brake control device 94, a steering control device 95, and the like via a bus 96.

[0067] The vehicle control device 90 is configured, for example, by a microcomputer that combines a CPU (Central Processing Unit) that executes calculations, a ROM (Read Only Memory) as a secondary storage device that records programs for the calculations, and a RAM (Random Access Memory) as a temporary storage device that saves the calculation progress and temporary control variables, and by executing the stored programs, it realizes each function of a recognition unit 91, a judgment unit 92, a control unit 93, etc.

[0068] The recognition unit 91 reads out the inference result of the inference unit 1 of the image processing device 100 that receives the captured image captured by the camera 99 as an input, and recognizes the external environment based on the inference result.

[0069] The determination unit 92 determines the action plan, driving route, etc. of the vehicle 900 based on the recognition result of the external environment by the recognition unit 91, and outputs the determination result to the control unit 93.

[0070] The control unit 93 calculates control command values ​​for the brake control device 94, the steering control device 94, etc. based on the judgment result by the judgment unit 92, and controls the vehicle 900 by outputting them to the brake control device 94, the steering control device 94, etc.

[0071] With this configuration, the server 500 communicates with many vehicles 900, distributes the desired accuracy improvement target image stored in the accuracy target image storage unit 32 to many vehicles, and uses a large number of captured images taken by the cameras 99 mounted on many vehicles as candidate images for learning. This allows for efficient re-learning of the desired image recognition model.

[0072] (Hardware configuration) Finally, a description will be given of the hardware configurations of the image processing device 100 and the server 500. Here, the hardware configurations will be described taking the image processing device 100 and the server according to the first embodiment as an example, but the same applies to the image processing devices according to the second embodiment, the third embodiment, the first modification, and the second modification.

[0073] FIG. 8 is a diagram illustrating an example of the hardware configuration of the image processing device 100. As shown in FIG.

[0074] The image processing device 100 realizes an image processing method in which each block cooperates with each other to perform the above-mentioned processes by causing a computer to execute a program.

[0075] The image processing device 100 includes a central processing unit (CPU) 200, a read only memory (ROM) 201, a random access memory (RAM) 202, a non-volatile storage 203, and a transmission / reception unit 204, all of which are connected to a bus 205.

[0076] The CPU 200 reads out program code of software for implementing each function according to the present embodiment from the ROM 201, loads it into the RAM 202, and executes it. Variables, parameters, etc. generated during the computation process of the CPU 200 are temporarily written into the RAM 202, and these variables, parameters, etc. are read out by the CPU 202 as appropriate. However, instead of the CPU 200, an MPU (Micro Processing Unit) or a GPU (Graphics Processing Unit) may be used, or the CPU 200 and a GPU (Graphics Processing Unit) may be used in combination. For example, the functions of the inference unit 1 and the learning image generation unit 2 are implemented by the CPU 200, the ROM 201, and the RAM 202.

[0077] The non-volatile storage 203 may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a flexible disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, or a non-volatile memory. In addition to an operating system (OS) and various parameters, a program for making the image processing device 100 function is recorded in the non-volatile storage 203. The ROM 201 and the non-volatile storage 203 record programs and data necessary for the CPU 200 to operate, and are used as an example of a computer-readable non-transient storage medium storing a program executed by the image processing device 100. For example, the functions of the learning candidate image group storage unit 31, the similarity threshold storage unit 32, the accuracy improvement target image storage unit 33, and the learning image storage unit 34 are realized by the non-volatile storage 203.

[0078] For example, a NIC (Network Interface Card) or the like is used for the transmitting / receiving unit 508, and various data can be transmitted and received between devices via wireless communication connected to a terminal of the NIC. Also, the device is configured to be capable of transmitting and receiving various data to and from a server or the like via a network such as a LAN or the Internet, or a dedicated line. The server 500 also has a similar hardware configuration, and a computer executes a program to realize an information processing method in which each block cooperates to perform the above-mentioned processing.

[0079] Although the above-mentioned modified example 2 is an example applied to the recognition of an image captured by an in-vehicle camera, the present invention can also be applied to other image recognition devices. For example, the present invention can also be applied to a recognition device for images captured by a traffic volume survey camera, an image captured by a security camera, and an image captured by a camera for inspecting processed products in a factory.

[0080] The above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits, etc. Also, the above-mentioned configurations, functions, etc. may be realized in software, by a processor interpreting and executing a program that realizes each function.

[0081] The present invention is not limited to the above-mentioned embodiments, and various other applications and modifications are possible without departing from the gist of the present invention described in the claims. For example, the above-mentioned embodiments are detailed and specific descriptions of the configurations of the devices and systems in order to clearly explain the present invention, and are not necessarily limited to those having all the described configurations. In addition, it is possible to replace a part of the configuration of the embodiment described here with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of a certain embodiment. In addition, it is also possible to add, delete, or replace other configurations with respect to a part of the configuration of each embodiment. In addition, the control lines and information lines are those that are considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product. In reality, it may be considered that almost all configurations are connected to each other. [Explanation of symbols]

[0082] Inference unit 1, learning image generation unit 2, similarity calculation unit 21, similar image selection unit 22, memory unit 3, learning candidate image group memory unit 31, accuracy target image memory unit 32, similarity threshold memory unit 33, learning image memory unit 34, server 500, learning unit 510, server memory unit 520, model storage unit 530, image processing device 100

Claims

1. An inference unit that inputs the target image and the candidate image group into an image recognition model, and based on the output of the intermediate layer of the image recognition model obtained by inference, determines feature quantities related to the target image and feature quantities related to the candidate images included in the candidate image group, and outputs the feature quantities. A training image generation unit generates training images to be used for training the target image, according to the feature similarity, which is the similarity between the feature quantities relating to the target image and the feature quantities relating to the candidate image, output from the inference unit. An image processing apparatus characterized by comprising:

2. In the image processing apparatus according to claim 1, The aforementioned training image generation unit outputs the feature similarity and the generated training image. An image processing apparatus comprising a generation quantity adjustment unit that determines the amount of training images to be generated based on the feature similarity output from the training image generation unit and the training images.

3. In the image processing apparatus according to claim 1, The learning image generation unit includes a similar image selection unit that selects similar images from the candidate images included in the candidate image group whose feature similarity with the target image exceeds a threshold. An image processing apparatus characterized by generating the training image using the similar image selected by the similar image selection unit.

4. In the image processing apparatus according to claim 2, The image processing apparatus is characterized in that the generation quantity adjustment unit reduces the amount of training images generated as the feature similarity increases.

5. In the image processing apparatus according to claim 1, The inference unit outputs the feature quantities and further outputs the result of the inference as the confidence level of the target image. The aforementioned training image generation unit outputs the feature similarity and the generated training image. An image processing apparatus comprising a generation quantity adjustment unit that takes the feature similarity output from the training image generation unit, the training image, and the confidence level output from the inference unit as input, and determines the amount of training images to be generated based on the feature similarity and the confidence level.

6. In the image processing apparatus according to claim 5, The image processing apparatus is characterized in that the generation quantity adjustment unit reduces the amount of training images generated as the reliability increases.

7. In the image processing apparatus according to claim 1, The inference unit is characterized by performing recognition of an image captured by an in-vehicle camera.

8. An image processing system comprising an image processing device according to claim 1 and a server communicated with the image processing device via a network, The aforementioned server, A learning unit that performs training of the image recognition model based on the aforementioned training images, An image processing system characterized by comprising a model storage unit for storing a trained model.

9. Based on the output of the intermediate layer of the image recognition model obtained by inference by inputting the target image and the candidate image group into the image recognition model, feature quantities related to the target image and feature quantities related to the candidate images included in the candidate image group are obtained and the feature quantities are output. An image processing method that generates training images to be used for training the target image, according to the feature similarity, which is the similarity between the outputted feature quantities relating to the target image and the feature quantities relating to the candidate image.

Images