Learning data generating method, learning data generating apparatus, and learning data generating system

By selecting images near cluster boundaries using multiple AI models with different characteristics and generating similar images, the method enhances the effectiveness of training data for AI models, addressing the issue of model-specific weak image determination.

JP2024158171APending Publication Date: 2024-11-08DENSO TEN LTD

Patent Information

Application Number
JP2023073142
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-27
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing methods for determining weak images for AI models are not model-specific, leading to ineffective training data sets that may not improve judgment accuracy, as images deemed weak by one model may not be so for another.

Method used

A method that selects learning data by identifying images near cluster boundaries using multiple AI models with different characteristics, focusing on regions of interest with significant changes in feature amounts, and generating similar images to enhance training data.

Benefits of technology

This approach ensures that the selected images are effectively weak for each AI model, improving judgment accuracy and efficiency of learning processes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024158171000001_ABST
    Figure 2024158171000001_ABST
Patent Text Reader

Abstract

To provide learning data having a high learning efficiency to be used for learning an object detection AI model.SOLUTION: The method for generating learning data detects a region of interest based on the importance for a determination result in determining an object type for a learning data candidate image by a plurality of AI models having different characteristics, and selects the learning data candidate image as the learning data when the amount of change in the region of interest between the AI models is larger than a threshold value.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a learning data creation method, a learning data creation device, and a learning data creation system. [Background technology]

[0002] Conventionally, to create an AI (Artificial Intelligence) model for image recognition (image classification, object detection, etc.) using deep learning, a large amount of training teacher data, ranging from several thousand to several million images, labeled (object type, object range, etc.) is required. On the other hand, additional learning of already learned data, etc., may result in overlearning of the AI ​​model. In order to improve the judgment accuracy of the AI ​​model, it is effective and efficient to additionally learn images that the AI ​​model has difficulty judging by adding them to a learning dataset. As a method for judging difficult images, a conventional technology is known in which images near cluster boundaries are judged to be difficult images (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-21201 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, an image that is determined to be a difficult image in one AI model may not be determined to be a difficult image in another AI model. For this reason, when a method of determining difficult images in an AI model as in the conventional technology (for example, determining an image near a cluster boundary as a difficult image as described above) is used, the extracted difficult images may not be difficult images for the AI ​​model to be trained. In addition, in a training data set created for general-purpose use, a training data set based on difficult images created for a certain AI model may contain many images that are not difficult images for other AI models, and therefore cannot be used suitably for additional training. Therefore, it is necessary to determine difficult images for each AI model and create training data.

[0005] In view of the above problems, the present invention aims to provide a technique capable of creating suitable training data for a training target AI model. [Means for solving the problem]

[0006] An exemplary training data creation method of the present invention is a training data creation method that creates training data for training an AI model that determines object types, and in determining the object type of training data candidate images using a plurality of AI models with different characteristics, detects an area of ​​interest based on its importance to the determination result, determines the amount of change in the area of ​​interest between the AI ​​models, and selects the training data candidate images as the training data if the amount of change is greater than a predetermined threshold. Effect of the Invention

[0007] If there is a difference between the AI ​​models in the attention area of ​​an object image that has a large effect on object judgment in different AI models, the object image can be estimated as a difficult image for each AI model. The present invention utilizes this characteristic to select a difficult image that has a high learning effect in an AI model. Therefore, the present invention can contribute to the creation of learning data that can be used for efficient learning. [Brief description of the drawings]

[0008] [Figure 1] A configuration diagram showing an example of a learning data creation system. [Figure 2A] FIG. 1 is an explanatory diagram showing an example of a surrounding image captured by a vehicle; [Figure 2B] FIG. 1 is an explanatory diagram showing an example of an object detection result in an ambient image captured by a vehicle; [Diagram 3] FIG. 1 is an explanatory diagram showing an example of a data scatter diagram of feature amounts for a detected object; [Figure 4A] FIG. 1 is an explanatory diagram showing a first example of an attention region for object detection in a poor candidate image; [Figure 4B] FIG. 11 is an explanatory diagram showing a second example of an attention area related to object detection of a poor candidate image; [Diagram 5] FIG. 13 is an explanatory diagram showing an example of a method for generating a similar image similar to a poor image; [Figure 6] A flowchart showing the difficult candidate image processing executed by the in-vehicle learning device of FIG. 1. [Figure 7] A flowchart showing a learning data creation process executed by the learning data creation device of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the contents of the embodiments described below.

[0010] Fig. 1 is a configuration diagram showing an example of a learning data creation system. Fig. 1 shows components necessary for explaining the features of this embodiment, and omits the description of general components.

[0011] The learning data creation system 1 includes an on-board learning device 10 of a vehicle V, and a learning data creation device 40 configured as a server device or the like. From the viewpoint of collecting a large amount of data, a learning data creation system consisting of the learning data creation device 40 and the on-board learning devices 10 of a plurality of vehicles is preferable. The on-board learning device 10 and the learning data creation device 40 are connected to each other so as to be able to communicate bidirectionally via a communication network (not shown) such as a mobile communication network.

[0012] The vehicle V is equipped with a camera C, an on-board learning device 10, and a vehicle control device 50. The camera C captures images of the surroundings of the vehicle V. The camera C is installed, for example, in the front, rear, left and right sides of the vehicle V, and is composed of a front camera, a rear camera, a left camera, and a right camera that capture images in these directions.

[0013] The vehicle control device 50 notifies the driver of the presence of an obstacle (position, object type, etc.) based on the result of object (obstacle) detection around the vehicle using an in-vehicle AI model described later, and also performs vehicle control such as automatic braking control. In other words, the in-vehicle AI (object detection function using an in-vehicle AI model) is used for various vehicle controls and also for generating learning data described later.

[0014] The on-board learning device 10 is configured as a so-called computer device including a communication unit 11, a storage unit 12, and a controller 13. The on-board learning device 10 performs learning processing of an on-board AI model 123, which will be described later.

[0015] The communication unit 11 is an interface for communicating data with other devices (devices in the vehicle V and external devices) via a communication network. The communication unit 11 includes a wireless communication device for wirelessly communicating with the learning data creation device 40, and is configured, for example, by a transmission / reception device of a mobile telephone network in 5G communication (fifth generation mobile communication system).

[0016] The storage unit 12 includes a volatile memory and a non-volatile memory, and stores various information necessary for various processes such as the learning process of the in-vehicle AI model 123. The volatile memory is, for example, a random access memory (RAM). The non-volatile memory is, for example, a read only memory (ROM), a flash memory, and a hard disk drive. The non-volatile memory stores programs and data that can be read by the controller 13. At least a part of the programs and data stored in the non-volatile memory may be obtained from another computer device (server device) connected by wire or wirelessly, or from a portable recording medium.

[0017] The storage unit 12 stores image data 121, a detection processing program 122, and an in-vehicle AI model 123. The image data 121 is data of an image of the surroundings of the vehicle V captured by the camera C.

[0018] The detection processing program 122 is a program related to object detection executed by the controller 13. The detection processing program 122 includes programs for various functions related to object detection that are executed and realized by the controller 13.

[0019] The in-vehicle AI model 123 is an AI model for image recognition that performs image classification, object detection, region extraction, etc. on the image data 121, and is, for example, an AI model that uses a convolutional neural network (Region with Convolutional Neural Networks, R-CNN) for object detection and region detection.

[0020] The in-vehicle AI model 123 is an AI model that has been trained using a separate appropriate training data set (in this example, a training data set of road surroundings images (object types in the images are taken as correct answer data)). The in-vehicle AI model 123 is capable of detecting features for a detected object, similar to a general AI model, that is, capable of detecting detection target objects (vehicles and traffic lights in the image examples described below) that exist around the road. The in-vehicle AI model 123 also outputs data on the features for each detected object. Furthermore, similar to a general AI model, the in-vehicle AI model 123 detects (calculates) the presence area of ​​the detected object in the image, that is, a so-called bounding box.

[0021] The controller 13 is composed of a processor that performs arithmetic processing and the like, and controls various operations in the on-board learning device 10. The processor includes, for example, a CPU (Central Processing Unit). The controller 13 operates the on-board AI model 123 to realize various functions. In other words, the on-board AI model 123 is a type of program, and the controller 13 realizes the functions as an AI model.

[0022] The functions of the controller 13 include an acquisition unit 131, a detection unit 132, and a grouping unit 133. In this embodiment, the functions of the controller 13 are realized by the processor executing arithmetic processing in accordance with a detection processing program 122 stored in the storage unit 12.

[0023] The acquisition unit 131 acquires an image of the surroundings of the vehicle V captured by the camera C of the vehicle V. The acquisition unit 131 stores in the storage unit 12 image data 121 of the acquired surrounding image.

[0024] The detection unit 132 performs object detection on the surrounding image (image data 121) captured by the vehicle V. In detail, the detection unit 132 uses an in-vehicle AI model 123 using R-CNN or the like to set an object region (bounding box) including each object included in the surrounding image and classify the objects (feature values) in each object region.

[0025] Fig. 2A is an explanatory diagram showing an example of a surrounding image captured by a vehicle V. Images 21 and 22 shown in Fig. 2A are examples of surrounding images (image data 121) captured by the vehicle V. Image 21 includes two objects 211 and 212. Image 22 includes two objects 221 and 222. Objects 211 and 221 are traffic lights, and objects 212 and 222 are passenger cars.

[0026] Fig. 2B is an explanatory diagram showing an example of an object detection result in a surrounding image captured by a vehicle V. In an image 21 shown in Fig. 2B, rectangular regions 211R and 212R surrounded by two dashed lines are object regions detected by the detection unit 132 and set in the image 21. The object regions 211R and 212R include objects 211 and 212, respectively. In an image 22 shown in Fig. 2B, rectangular regions 221R and 222R surrounded by two dashed lines are object regions detected by the detection unit 132 and set in the image 22. The object regions 221R and 222R include objects 221 and 222, respectively.

[0027] The detection result data for object detection on image 21 includes information identifying the positions of object regions 211R, 212R in image 21 as position information of objects 211, 212, and also includes object information indicating that objects 211, 212 are classified as a traffic light and a passenger vehicle, respectively.

[0028] The detection result data for object detection for image 22 includes information identifying the positions of object regions 221R, 222R in image 22 as position information of objects 221, 222, and also includes object information indicating that objects 221, 222 are classified as a traffic light and a passenger vehicle, respectively.

[0029] The grouping unit 133 acquires information related to object detection processing (feature amounts in processing such as R-CNN) included in the surrounding images detected by the detection unit 132, and performs clustering on the feature amounts. The grouping unit 133 performs clustering based on the feature amounts of the surrounding images using a clustering algorithm represented by, for example, the k-means method (k-means clustering).

[0030] Fig. 3 is an explanatory diagram showing an example of a data scatter diagram of features (an example in the case of a two-dimensional feature vector) for a detected object. The horizontal and vertical axes in Fig. 3 respectively indicate features (X and Y values ​​in a feature vector). In Fig. 3, elliptical regions 31, 32, 33, and 34 surrounded by four dashed lines are so-called cluster regions, regions in which objects with feature vectors within the region range are determined to be the same type of object (passenger car, traffic light, etc.). The four dashed lines themselves indicate cluster boundaries in each cluster region.

[0031] These cluster regions are set based on, for example, the distribution state of the features for each detected object collected as a sample for cluster setting (e.g., a group of feature plot points with high density is set as one region). Also, a method is possible in which the cluster regions are corrected based on the object detection results (feature vector distribution) after the cluster regions are set. Then, it is determined in which cluster region the detected object is located based on its features, and the detected object is estimated to be of the object type corresponding to that cluster region.

[0032] For example, object 211 (traffic light) and object 212 (car) detected by detection unit 132 and classified in image 21 are grouped by grouping unit 133 into cluster 31 (a cluster including the feature vector of the traffic light) and cluster 32 (a cluster including the feature vector of the car). Then, object 211 and object 212 are estimated to be a traffic light and a car, respectively.

[0033] That is, an object whose feature value is located within the cluster boundary (region) of cluster 31 is an object whose image resembles a traffic light. An object whose feature value is located within the cluster boundary of cluster 32 is an object whose image resembles a passenger car. Furthermore, other types of objects are grouped by the grouping unit 133 into clusters 33, 34, etc., in which the feature values ​​of the corresponding objects are located.

[0034] For each of the clusters 31, 32, 33, and 34, the cluster centroids 31c, 32c, 33c, and 34c are calculated based on the feature coordinates (vector coordinates) of each image distributed within the cluster. Then, the grouping unit 133 estimates the likelihood that the object belongs to the cluster for each centroid based on the distance between the feature coordinates (vector coordinates) in the image of each object and each centroid. In other words, the longer the distance between the feature coordinates of the object to be determined and the centroid of the cluster, the lower the accuracy of the determination result as to whether the object belongs to the cluster (whether the object to be determined is of an object type corresponding to the cluster). In other words, it can be said that the image of the object is more difficult to recognize as the distance from the centroid increases.

[0035] In addition, based on such characteristics, the boundary of a cluster can be set based on the distance from the center of gravity of the cluster. For example, based on the distribution state of the feature for each detected object, the density of the plot points of the feature of the detected object is calculated for each distance from the center of gravity. Then, the distance at which the density is an appropriate density that is estimated to be the cluster boundary is set as the boundary distance (distance from the center of gravity) that is the threshold value of the cluster boundary. In this way, by comparing the distance between the feature (plot points) of the detected object and the cluster center of gravity with the above-mentioned boundary distance, it can be determined whether the feature of the detected object is located near the cluster boundary.

[0036] The grouping unit 133 sets an object image determined to be located near a cluster boundary within a cluster region as a poor candidate image for which the in-vehicle AI model 123 has difficulty determining the object type. Then, the grouping unit 133 stores the set poor candidate image in the storage unit 12 in association with the feature amount and the cluster in which the feature amount is located (the object type corresponding to the cluster).

[0037] 3, in cluster 31, object 221 (traffic light) is located near the cluster boundary within the cluster 31 region. Also, in cluster 32, object 222 (car) is located near the cluster boundary within the cluster 32 region.

[0038] Note that "near the cluster boundary" does not only refer to a detected object at the cluster boundary where the distance between the cluster center and the feature amount of the detected object is exactly the same as the boundary distance, but also refers to the vicinity of the cluster boundary where the accuracy of the determination result is as low as that of the object at the cluster boundary. The distance from the cluster boundary that is considered to be near the cluster boundary may be a value determined in advance by experiments or simulations.

[0039] The controller 13 transmits data of the poor candidate images determined by the grouping unit 133 to be located on the cluster boundary to the learning data creation device 40 via the communication unit 11 at a predetermined transmission timing set in advance.

[0040] As described above, by defining object images determined to be located near a cluster boundary within a cluster area in vehicle V as candidate images that are difficult for the in-vehicle AI model 123 to judge, it is possible to efficiently collect images for model training that are effective in improving the judgment accuracy of the in-vehicle AI model 123.

[0041] Returning to FIG. 1, the learning data creation device 40 is a server device installed in a data management center or the like. The server device may be a physical server or a virtual server. The learning data creation device 40 selects a weak image from the collected weak image candidates and creates learning data. The learning data creation device 40 includes a communication unit 41, a storage unit 42, and a controller 43.

[0042] The communication unit 41 is an interface for communicating data with other devices (devices within the learning data creation device 40 and external devices) via a communication network. The communication unit 41 includes a wireless communication device for wireless communication with the vehicle V, and is configured, for example, with a transmission / reception device of a mobile telephone network for 5G communication (fifth generation mobile communication system).

[0043] The storage unit 42 includes a volatile memory and a non-volatile memory, and stores various information necessary for the learning process of the learning model of image recognition. The volatile memory is, for example, a random access memory (RAM). The non-volatile memory is, for example, a read only memory (ROM), a flash memory, and a hard disk drive. The non-volatile memory stores programs and data that can be read by the controller 43. At least a part of the programs and data stored in the non-volatile memory may be obtained from another computer device connected by wire or wirelessly, or from a portable recording medium.

[0044] The memory unit 42 stores image data 421, an attention area identification program 422, a first learning model 4231, a second learning model 4232, ..., an nth learning model 423n for identifying an attention area, a similar image generation program 424, and a learning dataset 425.

[0045] The image data 421 is data of a poor candidate image of the vehicle V that is photographed by the camera C of the vehicle V and received via the communication unit 41. In detail, the image data 421 is data of a poor candidate image that is identified by the controller 13 of the on-board learning device 10 as being located at a cluster boundary, and the image and the feature amount are linked. The image data 421 includes data of poor candidate images identified in multiple vehicles V.

[0046] The attention area identification program 422 is a program related to identification of an attention area that is an area that has a high influence on object detection (determination) in the object detection process executed by the controller 43 and has high importance as a basis for judgment. The attention area identification program 422 is composed of algorithms such as LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations). In other words, the attention area is an area that is a main basis for object type determination and has characteristics unique to the object type. The attention area identification program 422 realizes various functions related to the identification of the attention area that are executed and realized by the controller 43.

[0047] The first learning model 4231, the second learning model 4232, ..., the n-th learning model 423n for identifying an attention area are learning models for identifying an attention area in the object type determination process for the image data 421, and are learning models to which an attention area identification program is applied. Each of these learning models 423n has different characteristics, and is different in model configuration, various operation parameters, etc. Also, each of these learning models 423n is composed of an AI model, etc. It is preferable that the learning model 423n for identifying an attention area identifies an area that the on-board AI model 123 of the on-board learning device 10 of the vehicle V regards as an attention area when determining the object type as the attention area. For this reason, it is preferable that the learning model 423n includes a model corresponding to the algorithm (particularly the object type determination algorithm part) of the on-board AI model 123 of each vehicle.

[0048] The attention area identification program identifies an attention area as data calculated by an algorithm (LIME, SHAP, etc.) having a function of outputting the basis for the determination of object detection by each learning model 423n for identifying the attention area. By using such an algorithm, it is possible to grasp the area focused on as the basis for determining the type of object.

[0049] The similar image generating program 424 is a program related to the generation of a similar image executed by the controller 43. The similar image generating program 424 is executed by the controller 43 to realize various functions related to the generation of a similar image for the original image.

[0050] The learning data set 425 is a group of teacher data used for learning the AI ​​model. Similar images generated by the similar image generation program 424 are added to the learning data set 425 as learning data.

[0051] The controller 43 is configured with a processor that performs arithmetic processing and the like, and controls various operations in the training data creation device 40. The processor is configured to include, for example, a CPU. The controller 43 performs various processes such as generation processing of the training data set 425.

[0052] The controller 43 includes, as its functions, an acquisition unit 431, an identification unit 432, and a generation unit 434. In this embodiment, the functions of the controller 13 are realized by the processor executing arithmetic processing in accordance with various programs stored in the storage unit 12, such as an attention area identification program 422 and a similar image generation program 424.

[0053] The acquisition unit 431 acquires image data of the poor candidate images and their object type data (object type corresponding to the cluster area in which the images are located) determined in each of the multiple vehicles V (on-board learning devices 10) via the communication unit 41. The acquisition unit 431 stores the acquired image data in the storage unit 12 as image data 421.

[0054] The identification unit 432 identifies an attention area related to the basis for the judgment of object detection using multiple learning models for the acquired poor candidate image (image data 421). In detail, the identification unit 432 applies the poor candidate image (image data 421) to a first learning model 4231, a second learning model 4232, ..., an n-th learning model 423n for identifying an attention area, and identifies an area that was the basis for the judgment of the object type in the poor candidate image of each of the learning models 423n, that is, an attention area that has a large influence on the judgment result.

[0055] Fig. 4A is an explanatory diagram showing a first example of an attention area related to object detection of a poor candidate image. In detail, Fig. 4A is an explanatory diagram showing an attention area for an object 222 (car) on a poor candidate image (image data 421) detected by LIME. Fig. 4A shows a schematic depiction of which area of ​​the object image is important for object type determination of the object 222 (car) (which area of ​​the object image has a large influence on the determination result).

[0056] In LIME, an area of ​​interest related to the basis for object detection (an area of ​​interest that has a large influence on the determination result) is converted into a group of pixels (superpixels) having similar color information and visualized. Area 432a in FIG. 4A is an area that has a large influence on the object determination process, and is an area of ​​interest calculated based on the degree of influence (importance) of the basis for object detection determination. Identification unit 432 identifies an area of ​​interest related to the basis for object detection determination by LIME.

[0057] Fig. 4B is an explanatory diagram showing a second example of an attention area related to object detection of a poor candidate image. In detail, Fig. 4B is an explanatory diagram showing an attention area for an object 222 (car) on a poor candidate image (image data 421) detected by SHAP. In Fig. 4B, the left end is an input image, the center is an image showing an attention area in a correct judgment result (Car: passenger car), and the right end is an image showing an attention area in an incorrect judgment result (Bus: bus) with respect to the correct answer, and they are drawn in a schematic manner.

[0058] In SHAP, the attention area related to the basis for the judgment of object detection is converted into pixels having different hues and brightness and visualized. That is, in the display in SHAP, each area of ​​the poor candidate image is displayed in a display color of a hue or brightness according to the importance to the judgment result of object detection. The area 432b in FIG. 4B is an area with a high judgment importance (SHAP Value), and is an attention area related to the basis for the judgment of object detection. The identification unit 432 identifies an attention area related to the basis for the judgment of object detection by SHAP. Specifically, the identification unit 432 identifies a pixel part higher than a reference value such as an average value of the importance of the entire image as an attention area based on the SHAP Value, which is the importance of each pixel in the poor candidate image calculated by the attention area identification program. The reference value may be a value determined in advance by an experiment or simulation.

[0059] Using such a method for determining an attention area, the specification unit 432 specifies an attention area related to the determination basis of the object determination by each learning model 423n for the acquired poor candidate image (image data 421).

[0060] The selection unit 433 calculates the amount of change (difference in shape, size, and position) of the attention area related to the judgment of each learning model 423n for the poor candidate image. The selection unit 433 calculates the amount of change of the attention area by calculating the difference between each vector data constituting each attention area to be compared using a similarity calculation algorithm represented by cosine similarity, for example.

[0061] When the change amount of the attention area calculated for each learning model 423n is large, that is, when the variation of the attention area for object determination in the target image is large, the area with high importance to the determination result is not stable, and the variation of the object type determination is also large. Therefore, such a poor candidate image is called a poor image. Therefore, the selection unit 433 judges a poor candidate image with a large change amount of the attention area calculated for each learning model 423n as a poor image with a high learning effect, and selects it as learning data. In other words, when the calculated change amount of the attention area by each learning model 423n exceeds a predetermined judgment threshold, the selection unit 433 determines the poor candidate image to be judged as a poor image, and selects the poor image as learning data.

[0062] Specifically, the selection unit 433 replaces the importance calculated for each pixel of the poor candidate image with a linear matrix, and calculates a vector value by replacing the importance components outside the attention area with 0. The selection unit 433 also calculates cosine similarity from the vector value of one learning model and the vector value of another learning model, using the vector values ​​calculated in each learning model. Then, the selection unit 433 determines, as a poor candidate image, an image for which the average of cosine similarity in comparison with a plurality of learning models is lower than a preset threshold value (for example, 0.5). Note that although the importance components outside the attention area are replaced with 0, they may be replaced with the average importance value within the image.

[0063] As another possible method, an image comparison process can be used, for example, to calculate the amount of change (difference) in the area of ​​interest when determining the poor candidate image using image processing (detecting the edges of the processed image using LIME or SHAP and comparing the detected edge shapes and positions) to detect differences in the shape, size, and position of the area of ​​interest.

[0064] According to the above configuration, by using the attention area in object detection, it is possible to select images with variations in the area of ​​the judgment basis for object detection in multiple learning models from images acquired as poor candidate images. In other words, images with different attention areas in object detection in multiple learning models are likely to cause variations in the judgment results even for various object detection (type judgment) models, and are likely to be difficult images to judge. Therefore, it is possible to efficiently select poor images that are effective in improving the judgment accuracy in the learning model as image data for generating learning data.

[0065] The generating unit 434 generates a similar image similar to the difficult image selected by the selecting unit 433. In detail, the generating unit 434 generates a similar image having a feature amount of a predetermined range of similarity to the feature amount of the difficult image. The generating unit 434 generates a similar image similar to the difficult image using an AI algorithm represented by, for example, Generative Adversarial Networks (GAN).

[0066] FIG. 5 is an explanatory diagram showing an example of a method for generating a similar image 22S similar to a difficult image 22B. For example, by using a GAN, a similar image 22S that has been subjected to processing such as superimposing noise while maintaining the characteristics of the input difficult image 22B is generated. Depending on the design of the image processing (processing contents, strength adjustment, etc.), processing such as superimposing noise can be performed so that the similar image 22S is classified into the same cluster as the difficult image 22B from which it was generated. If it is not necessary to classify them into the same cluster, there is no need to use a GAN, and image processing such as simply superimposing noise based on a normal distribution around the area of ​​interest can also be applied.

[0067] Then, the generating unit 434 adds learning data, which uses the generated similar image 22S as input data and the object type of the original poor image 22B as correct answer data, to the learning data set 425. The original poor image 22B from which the similar image 22S was generated may also be added to the learning data set 425 as learning data. In this way, by adding the learning data of the similar image to the learning data set 425 and increasing the learning data of the learning data set 425, a learning effect can be expected and a highly versatile learning data set 425 can be obtained. In other words, it is possible to improve the judgment accuracy in a larger number of learning models.

[0068] In addition, by creating similar images using a generative adversarial network (GAN), it is possible to easily generate similar images that retain the characteristics of difficult images while being able to expect learning effects that are different from difficult images.

[0069] Fig. 6 is a flowchart showing the weak candidate image processing executed by the on-board learning device 10 of Fig. 1. This flowchart shows the technical contents of a computer program that causes a computer device to realize the processing performed by the on-board learning device 10. The computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program can also be provided from a server or the like using a communication network (so-called application by downloading). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0070] The process shown in Fig. 6 is executed by the controller 13 of the in-vehicle learning device 10. In this example, the process includes (serves as) a process for detecting an object (type) present in a captured image for use in vehicle control of each vehicle. This process is started when the ignition of the vehicle V is turned ON, and is repeatedly executed while the ignition of the vehicle V is ON.

[0071] In step S101, the controller 13 (acquisition unit 131) acquires an image of the surroundings of the vehicle V captured by the camera C of the vehicle V, and then proceeds to step S102.

[0072] In step S102, the controller 13 (the detection unit 132 and the grouping unit 133) performs object detection and clustering (object type determination) of the feature amounts related to the objects for the surrounding image of the vehicle V, and proceeds to step S103. This determination result is to be used separately for vehicle control.

[0073] In step S103, the controller 13 (grouping unit 133) determines whether the image data (feature amount) is an image located near a cluster boundary, and if it is an image located near a cluster boundary, proceeds to step S104, and if it is not located on a cluster boundary, returns to step S101.

[0074] In step S104, the controller 13 (grouping unit 133) sets images located near the cluster boundaries as candidate images that the in-vehicle AI model 123 has difficulty judging, associates the images, features, and object type (clustering result) with each other, and stores them in the memory unit 12, and then proceeds to step S105.

[0075] In step S105, the controller 13 determines whether it is time to transmit data related to the poor candidate image to the learning data creation device 40. If it is time to transmit, the process proceeds to step S106, and if it is not time to transmit, the process returns to step S101. The transmission timing may be the timing when communication with the server device is established. Also, for stable and high-speed communication, it may be the timing when the vehicle V is parked in a place where high-speed communication such as Wi-Fi (registered trademark) is available (such as a parking lot of a facility).

[0076] In step S106, the controller 13 transmits the data of the difficult candidate image to the learning data creation device 40 via the communication unit 11, and ends the process related to this flowchart.

[0077] Fig. 7 is a flowchart showing the learning data creation process executed by the learning data creation device 40 of Fig. 1. This flowchart shows the technical contents of a computer program that causes a computer device to realize the processes performed by the learning data creation device 40. The computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program can also be provided from a server or the like using a communication network (so-called application by downloading). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0078] The process shown in FIG. 7 is executed when a designer or the like of the learning data creation device 40 performs an operation to start the creation process using an operation unit such as a keyboard.

[0079] In step S201, the controller 43 (acquisition unit 431) acquires data (image data, object type data, etc.) of each difficult object candidate image determined in each of the multiple vehicles V via the communication unit 41, and proceeds to step S202.

[0080] In step S202, the controller 43 (identification unit 432) identifies an area of ​​interest related to the basis for determining object detection using multiple learning models, namely, the first learning model 4231, the second learning model 4232, ..., the nth learning model 423n, and proceeds to step S203.

[0081] In step S203, the controller 43 (selection unit 433) selects a difficult image from the difficult candidate images based on the amount of change in the attention area for each learning model, and the process proceeds to step S204.

[0082] In step S204, the controller 43 (the generation unit 434) generates a similar image from the poor image, and the process proceeds to step S205.

[0083] In step S205, the controller 43 (generation unit 434) adds learning data, which uses the generated similar image as input data and has the object type of the original difficult image as correct answer data, to the learning dataset 425, and ends the processing related to this flowchart.

[0084] Then, the learning data creation device 40 (controller 43) uses the learning data set 425 stored in the storage unit 42 to perform additional learning of the learning model. In this case, additional learning may be performed on the learning model 423n used for identifying the region of interest. The additional learning is performed using the learning data of the learning data set 425 and the correct answer data, using a known learning method such as an error backpropagation method. The parameters of the learning model 423n for identifying the region of interest are updated by the additional learning.

[0085] The learning model updated by the additional learning is transmitted to the vehicle V having the learning model before the update (the on-board AI model 123). The vehicle V replaces the on-board AI model 123 stored in the storage unit 12 with the received updated learning model.

[0086] <Notes, etc.> Various technical features disclosed as embodiments in this specification can be modified in various ways without departing from the spirit of the technical creation. In other words, the above embodiments are illustrative in all respects and are not restrictive. The technical scope of the present invention is indicated by the claims, not the description of the above embodiments, and includes all modifications that fall within the meaning and scope of the claims. In addition, the multiple embodiments shown in this specification may be combined as appropriate to the extent possible.

[0087] In the above embodiment, various functions are realized by software through the arithmetic processing of the CPU according to the program, but at least some of these functions may be realized by electrical hardware resources. Examples of hardware resources include ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays). Conversely, at least some of the functions realized by hardware resources may be realized by software.

[0088] The scope of the present embodiment may also include a computer program that causes a processor (computer) to realize at least some of the functions of the in-vehicle learning device 10 and the learning data creation device 40. The scope of the present embodiment may also include a computer-readable non-volatile recording medium that records such a computer program. The non-volatile recording medium may be, for example, the above-mentioned non-volatile memory, an optical recording medium (e.g., an optical disk), a magneto-optical recording medium (e.g., a magneto-optical disk), a USB memory, or an SD card. [Explanation of symbols]

[0089] 1. Learning data creation system 10. In-vehicle learning device 11 Communications Department 12 Storage section 13 Controller 40 Learning data creation device 41 Communications Department 42 Storage section 43 Controller 50 Vehicle control device 123 In-vehicle AI model 4231, 4232, 423n learning models

Claims

1. A learning data creation method for creating learning data for learning an AI model that determines an object type, comprising: In determining the object type of the learning data candidate images by the plurality of AI models having different characteristics, a region of interest is detected based on the importance of the region of interest to the determination result; Determining the amount of change in the region of interest between the AI ​​models; When the amount of change is greater than a predetermined threshold, the learning data candidate image is selected as the learning data. How to create training data.

2. In the classification of object types by the AI ​​model, the object image whose feature amount is located near a cluster boundary, which is a boundary between classifications of object types, is set as the learning data candidate image. The training data generating method according to claim 1 .

3. A similar image of the learning data is generated, and the generated similar image is used as the learning data. The training data generating method according to claim 1 .

4. The similar images are generated using a generative adversarial network. The training data generating method according to claim 3 .

5. A learning data creation device that creates learning data for learning an AI model that determines an object type, In determining the object type of the learning data candidate images by the plurality of AI models having different characteristics, a region of interest is detected based on the importance of the region of interest to the determination result; Determining the amount of change in the region of interest between the AI ​​models; When the amount of change is greater than a predetermined threshold, the learning data candidate image is selected as the learning data. Training data creation device.

6. A learning data creation system for creating learning data for learning an AI model that determines an object type, the learning data creation system including an in-vehicle device and a learning data creation device, The in-vehicle device includes: Acquire images captured by the vehicle-mounted camera, Controlling the vehicle based on an object detection result by the AI ​​model for the captured image; In classifying the captured image into object types by the AI ​​model, the object image having a feature amount located near a cluster boundary, which is a boundary between object types, is selected as a learning data candidate image; Transmitting the selected training data candidate image to the training data creation device; The learning data creation device includes: In determining the object type of the learning data candidate image by the plurality of AI models having different characteristics, a region of interest is detected based on the importance of the region of interest to the determination result; Determining the amount of change in the region of interest between the AI ​​models; When the amount of change is greater than a predetermined threshold, the learning data candidate image is selected as the learning data. Learning data creation system.

Citation Information

Patent Citations

  • Learning server, and assist system

    JP2019021201A

Cited By

  • Method for collecting training data for training ai model, and electronic device for performing same

    WO2026135197A1