Ai model training system and ai model training method

Abstract

To provide an AI model training system and AI model training method that can appropriately collect required recognition data when retraining an AI model.SOLUTION: An AI model training system includes: an AI inference processing unit 12A configured to perform inference processing on input data in an AI model; an AI verification unit 13A configured to acquire, as examination data, input data when the AI inference processing unit 12A has not correctly performed the inference processing; a feature amount extraction unit 13B configured to extract a feature amount in the examination data; a similar data collection unit 14A configured to collect similar data similar to the feature amount; and an AI training unit 15B configured to use the similar data collected by the similar data collection unit 14A to perform retraining on the AI model. The system extracts, from input data on which AI inference processing has not been performed correctly, a feature amount of the input data, and collects similar data from other data to perform retraining, thereby enabling appropriately collecting required recognition data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an AI model learning system and an AI model learning method for an automatic driving device that automatically drives a vehicle such as an automobile. [Background technology]

[0002] Recently, autonomous driving, which allows cars to drive autonomously, has been developed, and to execute autonomous driving, it is effective to use AI functions to control the driving of the car. In order to realize highly accurate inference control of the driving of the car using the AI ​​model installed in the car, it is important to verify and analyze the AI ​​model, especially the recognition model, using a sufficient amount of recognition data. Note that the recognition data is typically image data, and the following explanation will be given using image data as a representative example.

[0003] In order to ensure driving safety, the verification and analysis of this AI model requires the comprehensive collection of image data under various actual driving conditions over a huge amount of time. In particular, image data of the driving environment under a variety of driving conditions is necessary for autonomous driving of a car. For this reason, image data is collected while the car is actually driving, and verification and analysis are carried out.

[0004] However, in general, when collecting image data, a large amount of image data of scenes that are easy to photograph is collected, while image data of unusual scenes is not collected. As a result, various image data is not collected comprehensively, which means that sufficient verification and analysis cannot be carried out.

[0005] To resolve this situation, a technology for comprehensively collecting image data is known, for example, as described in Patent Document 1 (International Publication No. 2022 / 107595).

[0006] Patent Document 1 discloses that, for the purpose of efficiently relearning a recognition model, the system includes a collection timing control unit that controls the timing of collecting image data candidates that are candidates for image data to be used in relearning the recognition model, and an image data collection unit that selects image data from the collected image data candidates based on at least one of the characteristics of the image data candidates and the similarity with stored image data.

[0007] In addition, the recognition model is used to recognize a specified recognition target (hereinafter, sometimes referred to as a target object) around the vehicle, and the image data collection unit selects appropriate image data from image data candidates including image data obtained by photographing the surroundings of the vehicle with an imaging means such as a camera installed in the vehicle. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] International Publication No. 2022 / 107595 Summary of the Invention [Problem to be solved by the invention]

[0009] Incidentally, the method of collecting image data described in Patent Document 1 determines whether or not to collect image data based on a certain external event (e.g., an event such as the occurrence of construction work, the occurrence of an accident, rainfall conditions, or snowfall conditions), and classifies the image data collected based on this external event according to the output result (reliability) of the recognition model.

[0010] However, with this image data collection method, image data cannot be collected unless a corresponding external event occurs. Also, with this image data collection method, unnecessary image data is uniformly collected, resulting in unnecessary verification and analysis. In this context, there is a demand for appropriate collection of necessary image data.

[0011] An object of the present invention is to provide an AI model learning system and an AI model learning method that can appropriately collect the necessary image data in re-learning for verifying and analyzing an AI model. [Means for solving the problem]

[0012] The present invention is an AI model learning system for re-learning an AI model, characterized in having an AI inference processing unit that performs inference processing using an AI model on input data, an AI verification unit that acquires input data as consideration data when the inference processing in the AI ​​inference processing unit is not performed correctly, a feature extraction unit that extracts features in the consideration data, a similar data collection unit that collects similar data that is similar to the features, and an AI learning unit that re-learns the AI ​​model using the similar data collected by the similar data collection unit. Effect of the Invention

[0013] According to the present invention, for input data for which AI inference processing has not been performed correctly, features of the input data are extracted, and similar data similar to these features is collected from elsewhere and re-learning is performed, thereby making it possible to properly collect the required image data. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is an explanatory diagram for explaining the concept of an AI model learning system according to the present invention. [Diagram 2] FIG. 1 is a block diagram showing a configuration of an AI model learning system according to a first embodiment of the present invention. [Diagram 3] FIG. 2 is a flowchart showing a control flow of the AI ​​model learning system according to the first embodiment of the present invention. [Figure 4] FIG. 1 is an explanatory diagram illustrating an example of a recognition error when recognizing a target from image data using an AI model. [Diagram 5] FIG. 5 is a flow chart showing a first control flow for extracting an area of ​​the target recognition error shown in FIG. 4. [Figure 6] FIG. 5 is a flow chart showing a second control flow for extracting the region of the target recognition error shown in FIG. 4. [Figure 7] FIG. 5 is a flow chart showing a third control flow for extracting the region of the target recognition error shown in FIG. 4. [Figure 8] FIG. 11 is an explanatory diagram showing an example of extracting features from a feature extractor using a neural network; [Figure 9] FIG. 11 is a flowchart showing a control flow in a vehicle-side control device when searching for similar image data. [Figure 10] FIG. 11 is a control flowchart for determining the validity of similar image data from a vehicle-side control device. [Figure 11] FIG. 11 is a control flow chart showing a first control flow for selecting similar image data to be used for relearning. [Figure 12] FIG. 11 is a control flowchart showing a second control flow for selecting similar image data to be used for relearning. [Figure 13] FIG. 11 is a control flowchart showing a third control flow for selecting similar image data to be used for relearning. [Figure 14] FIG. 1 is a block diagram showing a configuration of an AI model learning system according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiment, and various modifications and application examples within the technical concept of the present invention are also included within its scope. EXAMPLES

[0016] 1 shows the configuration of an AI learning system for explaining the concept of a first embodiment of the present invention. In this embodiment, the system is composed of a server 10 equipped with a server-side control unit having an AI function unit, a vehicle (here, an automobile) equipped with a vehicle-side control device 11 also having an AI function unit, and communication means (not shown) provided in each of the server 10 and the vehicle so that they can communicate wirelessly.

[0017] In addition, the vehicle-side control device 11 has a function of searching for similar data that is similar (including matches) to input data that cannot be inferred by the AI ​​model and providing the similar data to the server 10, upon request from the server 10, as described below.

[0018] Here, the AI ​​function unit of the server 10 is equipped with an AI model (here, a recognition model) to be verified. Also, the AI ​​function unit of the vehicle-side control device is a current AI function unit capable of automatically driving the vehicle. The recognition data here is image data used to recognize predetermined recognition objects (targets, etc.) around the vehicle. The recognition objects are targets such as other vehicles, pedestrians, bicycles, and traffic signs around the vehicle, which are required for automatic driving.

[0019] Now, the vehicle-side control device 11 transmits image data that may have been recognized incorrectly by its own AI model to the server 10 via communication means. Note that the image data at this time is positioned as a candidate for recognition error. The server-side control unit of the server 10 has at least two functions: an "evaluation and verification function unit" and a "relearning function unit."

[0020] The "Evaluation and Verification Function Unit" is equipped with model analysis and model visualization functions, and is configured to allow developers and designers to verify the analysis results on screen. In addition, in the model analysis, verification is performed using an AI model on the received image data, and the AI ​​model extracts (extracts areas) scenes where targets in the image data cannot be correctly inferred.

[0021] For example, image 10A shows a deviation in the gaze area. Images (IMG.1 to IMG.3) show the deviation of the inferred value (shown by the solid line) from the correct value (shown by the dashed line) of the gaze area, with image (IMG.1) showing a small deviation and a certainty of 0.92. Image (IMG.2) showing a large deviation and a certainty of 0.51, and image (IMG.3) showing a medium deviation and a certainty of 0.76. For this reason, for example, image (IMG.2) is deemed to have an inaccurate inference of the gaze area.

[0022] Then, the feature amount of the scene that cannot be correctly inferred is extracted, and the data of this feature amount is transmitted to the vehicle-side control device 11 of many other vehicles. Here, the other vehicle-side control device 11 is not geographically restricted, and the data is transmitted to vehicles existing within the country, for example. This makes it possible to expand the range in which image data similar to the feature amount of the scene that cannot be correctly inferred exists.

[0023] Meanwhile, many other vehicle-side control devices 11 search for image data (hereinafter referred to as similar image data) similar to the features sent from among the features of the image data they hold, and if similar image data is found, they transmit the similar image data again to the server-side control unit of the server 10 via the communication means.

[0024] The "relearning function unit" of the server-side control unit executes the re-learning function using the received similar image data. The "relearning function unit" infers the collected similar image data using an AI model, and if it satisfies the teacher information, it confirms it as learned data. Here, the teacher information is provided by a developer or designer. When the re-learning is completed, an update of the AI ​​model is executed for the vehicle-side control device 11. The AI ​​model is updated as necessary, and can also be reflected in the next autonomous driving device as an upgraded version of the AI ​​model.

[0025] Here, the "evaluation and verification function unit" and the "relearning function unit" may be provided in the same server 10, or may be provided separately in different servers 10. For example, the "evaluation and verification function unit" may be provided in the server 10 of a parts manufacturer that manufactures the vehicle-side control device 11, and the "relearning function unit" may be provided in a vehicle manufacturer that manufactures the vehicle.

[0026] In this way, for image data where AI inference processing has not been performed correctly, feature values ​​of the image data are extracted, and similar image data that has similar features is collected from other vehicles and re-learned, making it possible to properly collect the required image data.

[0027] Next, the specific configuration and operation of the concept of the present embodiment shown in Fig. 1 will be described below with reference to the drawings. In the following description, the "evaluation and verification function unit" and the "relearning function unit" are described as being provided in the same server 10.

[0028] 2 shows the control block of the AI ​​model learning system, which is constructed in the server-side control unit of the server 10. The control block is composed of hardware such as a computer and a logic circuit, and software that executes computer operations, etc. For this reason, the control block can be considered as a functional block.

[0029] 2, the AI ​​model learning system of this embodiment is characterized by having an AI inference block 12 that performs inference processing on input image data using a recognition model, an AI verification block 13 that verifies the correctness of the inference processing using this recognition model, a learning data candidate collection block 14 that collects learning data candidates that are similar image data, and an AI learning block 15 that re-learns the AI ​​using the learning candidate data collected by the learning data candidate collection block 14. Note that "block" can also be read as "functional unit".

[0030] The AI ​​inference block 12 includes at least an AI inference processing unit 12A, an AI inference defect candidate detection unit 12B, and an input data acquisition unit 12C.

[0031] The AI ​​inference processing unit 12A infers, for example, candidates for image data that are likely to be targets from among the input image data. This image data is sent to the AI ​​inference defect candidate detection unit 12B, which detects image data for which the inference process may not have been performed correctly. The input data acquisition unit 12C acquires image data for which the inference process may not have been performed correctly, and inputs it to the AI ​​verification block 13.

[0032] The AI ​​verification block 13 includes at least an AI verification unit 13A and a feature extraction unit 13B.

[0033] The AI ​​verification unit 13A executes a judgment as to whether or not the target of the image data for which the inference process may not have been performed correctly is different from the correct answer value, whether or not it satisfies the threshold value, etc. Here, the correct answer value and the threshold value are set by the developer or the designer. Then, when it is judged that it is different from the correct answer value or does not satisfy the threshold value, the area where the target exists is set as the feature extraction area.

[0034] Next, the feature extraction unit 13B extracts the feature of the target from the image data of the set feature extraction region. The feature extraction is performed in the same manner as a well-known method in image processing. The feature extraction unit 13B inputs the extracted feature data to the learning data candidate collection block 14.

[0035] The learning data candidate collection block 14 includes at least a similar image data collection section 14A.

[0036] The similar image data collection unit 14A wirelessly transmits (e.g., push transmission or scheduled transmission) at least the feature data from the feature extraction unit 13B and search request data for similar image data having features similar to the feature data to other vehicles equipped with an automatic driving device. Here, the other vehicles are not geographically restricted and are vehicles that exist in a wide area (e.g., city unit, prefecture unit, and the whole of Japan).

[0037] This increases the possibility that similar image data exists, and also increases the speed of response since similar image data can be obtained from a database stored in each vehicle.

[0038] The vehicle-side control device 11 mounted on the vehicle searches for image data having features similar to the transmitted feature data from the image data stored in its own storage means (database), and if the corresponding image data is found, it wirelessly transmits a reply to the similar image data collection unit 14A. If the corresponding image data is not found, it wirelessly transmits reply data to that effect to the similar image data collection unit 14A. There may be one similar image data or multiple similar image data. The similar image data collection unit 14A transmits the collected similar image data to the AI ​​learning block 15 as learning image data.

[0039] The AI ​​learning block 15 includes at least an image selection unit 15A, an AI learning unit 15B, and an AI update unit 15C.

[0040] The image selection unit 15A selects the similar image data sent from the similar image data collection unit 14A as learning data, and this selection is performed by judging whether the similar image data is different from the correct answer value or whether it satisfies a threshold value. Here, the correct answer value and the threshold value are set by a developer or a designer. Then, if it is judged that the similar image data is different from the correct answer value or does not satisfy the threshold value, the similar image data is sent to the AI ​​learning unit 15B as learning data.

[0041] The AI ​​learning unit 15B executes well-known re-learning using the selected similar image data. After that, the AI ​​update unit 15C updates the AI ​​model (here, the recognition model). The updated AI model is fed back to the AI ​​inference processing unit 12A of the AI ​​inference block 12, and the AI ​​model of the AI ​​inference processing unit 12A is updated.

[0042] In addition, when image data is input from the AI ​​inference block 12 to the AI ​​verification block 13, the vehicle-side control device 11 is equipped with an image data storage function unit that stores the image data to be input to the AI ​​verification block 13 within the vehicle, provided that prior consent of the user is obtained, and when the AI ​​model of the vehicle-side control device 11 is updated by an AI model re-learned in the AI ​​learning block 15, an improvement effect confirmation function unit is provided that enables the user to confirm the improvement effect of the updated AI model on the image data stored in the vehicle.

[0043] The operation of the control block explained above is actually executed by the arithmetic processing of the computer constituting the server side control unit. Next, the control flow of the re-learning of similar image data by the computer will be explained with reference to FIG.

[0044] <Step S10> In step S10, AI inference is performed. This inference infers candidates for image data that are likely to be targets from the input image data. Then, image data for which the inference process may not have been performed correctly is detected from this image data. When image data for which the inference process has not been performed correctly is detected, the processes of steps S11 to S12 are executed.

[0045] <Step S11> ~ <Step S12> In steps S11 to S12, a determination is made as to whether or not the target object in the image data for which the inference process may not have been performed correctly differs from the correct answer value and whether or not the threshold value is satisfied. Here, the correct answer value and the threshold value are set by a developer or a designer.

[0046] If it is determined that the result is different from the correct answer or does not satisfy the threshold, it is determined that improvement is necessary, and the region in which the target exists is set as the feature extraction region. The AI ​​verification steps of steps S11 to S12 will be described in detail later.

[0047] Once the feature extraction region has been set, the process proceeds to step S13. If it is determined in step S12 that no improvement is required, the process goes to the end and waits for the next activation timing.

[0048] <Step S13> In step S13, feature amounts of the target object for which the inference process may not have been performed correctly are extracted from the image data of the feature amount extraction area set in step S12. This feature amount extraction is similar to a well-known method in image processing. After the feature amount is extracted, the process of step S14 is executed.

[0049] <Step S14> In step S14, the vehicle-side control device 11 of many other vehicles is requested to collect similar image data. This request is executed by wirelessly transmitting (for example, push transmission or scheduled transmission) the feature data obtained in step S13 and search request data for similar image data having features similar to the feature data to other vehicles equipped with an automatic driving device. Details of the search process of the vehicle-side control device 11 will be described later.

[0050] After the request in step S14, if similar image data is returned from the vehicle-side control device 11, the process of step 15 is executed. Since similar image data is expected to be returned from many other vehicle-side control devices 11, the received similar image data may be stored and step S15 may be executed at a predetermined time.

[0051] <Step S15> In step S15, the collected similar image data is analyzed to determine whether it is appropriate image data, and appropriate similar image data is selected. This selection is performed by determining whether the similar image data is different from a correct answer value and whether it satisfies a threshold value. The correct answer value and the threshold value are set by a developer or a designer. If it is determined that the data is different from the correct answer value or does not satisfy the threshold value, it is treated as learning data. The similar image data selection process in step S15 will be described in detail later. Once the similar image data is selected, the processes of steps S16 to S17 are executed.

[0052] <Step S16> to <Step S17> In step S16, the selected similar image data is used to execute well-known re-learning. When executing the re-learning, the recognition result of the target is re-verified in step S17. If the re-verification shows that the recognition was not correct and the inference result has not been improved, the process returns to step S14 and executes the same processing steps. On the other hand, if the re-verification shows that the recognition was correct and the inference result has been improved, the process exits and waits for the next start timing.

[0053] The above is the configuration of the AI ​​model learning system in this embodiment. Next, the main processing functions in the above-mentioned control flow will be explained.

[0054] First, a specific judgment method for the verification steps S11 to S12 will be described with reference to Fig. 4 to Fig. 7. Fig. 4 visually shows a state in which input image data is inferred and set in a feature extraction region, Fig. 5 shows a process flow for judging the need for improvement based on inference results, Fig. 6 shows a process flow for judging the need for improvement based on reliability, and Fig. 7 shows a process flow for judging the need for improvement based on judgment grounds.

[0055] In Fig. 4, input image data IMG-A input at a certain time shows vehicles CA, CB, and CC as targets toward the back of the screen. Similarly, a bicycle BY is present in the foreground of the screen. Also, a road marking RM indicating a right turn is drawn on the road.

[0056] When inference processing is performed using an AI model on such input image data, image data IMG-B shows the output state of the inference result for the target, and since vehicles CA, CB, and bicycle BY are recognized, the inference is correct, but vehicle CC may not be inferred correctly. Therefore, new image data is required to correctly infer vehicle CC.

[0057] Furthermore, when inference processing is performed using an AI model, image data IMG-C indicates the output state of reliability, and vehicle CA and bicycle BY are correctly inferred because their reliability exceeds a threshold (here, set to 0.50), but vehicle CB may not be correctly inferred because its reliability does not exceed the threshold. For this reason, new image data is required to correctly infer vehicle CB. Vehicle CC is not correctly inferred in the inference results, and is therefore undetected.

[0058] In addition, when inference processing is performed using an AI model, image data IMG-D shows the output state of the judgment grounds for the targets, and the judgment grounds for vehicles CA, CB, and CC correspond to the respective vehicles, but for bicycle BY, the judgment grounds are output for road marking RM, and it is possible that the judgment grounds have not been inferred correctly. For this reason, new image data is needed to correctly infer the judgment grounds for bicycle BY.

[0059] Next, a method for deriving each of the above-mentioned inference results will be described using a control flow. Figures 5 to 7 show a process flow for determining the necessity of improvement based on the inference results.

[0060] 5 is a flow chart showing a process for determining the inference result of a target object. Each process step will be explained below.

[0061] <Step S20> In step S20, image data for which there is a possibility that inference processing has not been performed correctly is input from the input data acquisition unit 12C. Once the image data has been input, the process of step S21 is executed.

[0062] <Step S21> In step S21, target inference processing is performed based on the input image data. This inference infers the "class information" of the target and the "position information" of the target. The "class information" is information on whether the target is a vehicle, motorcycle, pedestrian, etc., and the "position information" is coordinate information of the area surrounding the target. After the target inference processing is performed, the process of step S22 is performed.

[0063] <Step S22> In step S22, the inference result of the target inferred in step S21 is output. This output of the inference result is executed for all targets in the image data. When the inference result is output, the process of step S24 is executed. However, the process of step S23 is also executed in parallel.

[0064] <Step S23> In step S23, a designer or developer generates an inference result correct value for each target. The inference result correct value is the class information of the target and the position information of the target, and corresponds to the "class information" and "position information of the target" inferred in step S21. The generated inference result correct value is compared with the class information of the target and the position information of the target inferred in step S24.

[0065] <Step S24> ~ <Step S25> In step S24, the inference result of the target of the image data output in step S22 is compared with the correct inference result value of the target generated in step S23.

[0066] As a result of the comparison, if the inference result and the inference result correct value match (NO judgment), the process goes to END, and if they do not match (YES judgment), the process executes the processing of step S26. Returning to the image data IMG-B in Fig. 4, the vehicles CA, CB, and bicycle BY are targets whose inference result and the correct value match, and the vehicle CC is a target whose inference result and the correct value do not match.

[0067] <Step S26> In step S26, the mismatch area of ​​the target determined to be a mismatch is extracted. This mismatch area becomes the feature extraction area described above. As shown in image data IMG-B in FIG. 4, the vehicle CC is the target where the inference result and the correct answer value do not match. Then, the feature extraction unit 13B extracts features from the image data of the extracted mismatch area. When the mismatch area is extracted, the process ends.

[0068] 6 is a flow chart showing a process for determining the reliability of a target. Each process step will be explained below.

[0069] <Step S30> In step S30, image data for which there is a possibility that inference processing has not been performed correctly is input from the input data acquisition unit 12C. Once the image data has been input, the process of step S31 is executed.

[0070] <Step S31> In step S31, target inference processing is performed based on the input image data. This inference infers the "reliability" of the target. Here, the "reliability" is numerical information indicating the degree of confidence with which the inference was made. After the target inference processing is performed, the processing of step S32 is performed.

[0071] <Step S32> In step S32, the reliability of the target inferred in step S31 is output. This reliability output is performed for all targets in the image data. Once the reliability is output, the process of step S34 is executed. However, the process of step S33 is also executed in parallel.

[0072] <Step S33> In step S33, a reliability threshold is set by a designer or developer. Here, the reliability threshold is set to a lower limit of the reliability when recognizing a target included in image data (here, set to 0.50). Note that the reliability threshold can also be set for each target. The set reliability threshold is compared with the reliability of the target output in step S32.

[0073] <Step S34> ~ <Step S35> In step S34, the reliability of the target in the image data output in step S32 is compared with the target reliability threshold value generated in step S33.

[0074] As a result of the comparison, if the reliability in step S32 exceeds the reliability threshold (NO judgment), the process goes to end, and if it is below the reliability threshold (YES judgment), the process of step S36 is executed. Returning to IMG-C in Fig. 4, the vehicle CA and the bicycle BY have reliability of 0.50 or more, so they are targets whose reliability exceeds the reliability threshold, and the vehicle CB has a reliability of 0.45, so it is a target whose reliability is below the reliability threshold.

[0075] <Step S36> In step S36, a region of a target whose reliability is below the reliability threshold is extracted. This region becomes the feature extraction region described above. As shown in IMG-C in FIG. 4, the vehicle CB becomes a target whose reliability is below the reliability threshold. Then, the feature extraction unit 13B extracts features from the image data of the extracted region. When a region whose reliability is below the reliability threshold is extracted, the process ends.

[0076] 7 is a flow chart showing a process for determining an area to be noted (a determination basis area) for recognizing a target object. Each process step will be explained below.

[0077] <Step S40> In step S40, image data for which there is a possibility that inference processing has not been performed correctly is input from the input data acquisition unit 12C. Once the image data has been input, the process of step S41 is executed.

[0078] <Step S41> In step S41, target inference processing is performed based on the input image data. This inference infers the "judgment basis area" of the target. Here, the "judgment basis area" is information on an area to be focused on in order to recognize the target. After the target inference processing is performed, the processing of step S42 is performed.

[0079] <Step S42> In step S42, the judgment grounds area of ​​the target inferred in step S41 is output. This judgment grounds area is the area that was focused on when the target was inferred by inference. This output of the judgment grounds area is performed for all targets in the image data. Once the judgment grounds area is output, the process of step S44 is executed. However, the process of step S43 is also executed in parallel.

[0080] <Step S43> In step S43, a designer or developer sets a judgment basis area for each target. Here, the judgment basis area is set for all targets included in the image data. The set judgment basis area is compared with the judgment basis area of ​​the target output in step S42.

[0081] <Step S44> ~ <Step S45> In step S44, the determination basis region of the target in the image data output in step S42 is compared with the determination basis region of the target set in step S43.

[0082] As a result of the comparison, if the determination basis area in step S42 matches the set determination basis area (NO judgment), the process goes to END, and if it does not match (YES judgment), the process of step S46 is executed. Returning to image data IMG-D in Fig. 4, vehicles CA, CB, and CC are objects whose determination basis area in step S42 matches the set determination basis area, and bicycle BY is an object whose determination basis area in step S42 does not match the set determination basis area. In the case of bicycle BY, road marking RM is the determination basis area.

[0083] <Step S46> In step S46, the mismatched area of ​​the target determined to be a mismatch is extracted. This mismatched area becomes the feature extraction area described above. As shown in image data IMG-D in FIG. 4, the road marking RM and the bicycle BY become targets whose judgment basis areas do not match. Then, the image data of the extracted mismatched area has features extracted by feature extraction unit 13B. When the mismatched area is extracted, the process ends.

[0084] Then, the mismatched area of ​​the image data that needs improvement is treated as a feature extraction area, and the feature of the target is extracted. The feature may be extracted by various methods. For example, it is extracted using intermediate data when the image data corresponding to the feature extraction area is input to the AI ​​model. This state is shown in Figure 8.

[0085] In Fig. 8, the feature extraction unit of the AI ​​model is composed of a neural network consisting of a convolution layer and a pooling layer, and features can be extracted by passing the input image data through the convolution layer and the pooling layer in order. Note that the numbers of the image data input to the convolution layer in Fig. 8 indicate "horizontal pixel count x vertical pixel count x channel count." In this embodiment, the intermediate data before being input to the class identification unit 20 in particular is extracted as features by the region candidate extraction unit 21.

[0086] Next, a method for collecting similar image data similar to the extracted feature quantities will be described. When a request for collecting similar image data is transmitted from the server-side control unit to many other vehicles, many other vehicles will receive this transmission request. Then, upon receiving this collection request, the vehicle-side control unit 11 of each vehicle searches for similar image data from the image data stored and held by itself. The processing flow of this data search will be described with reference to FIG. 9.

[0087] <Step S50> In step S50, the system waits for a search request for similar image data from the server 10. In this state, the process of step S51 is executed.

[0088] <Step S51> In step S51, it is determined whether or not a search request has been received from the server 10. If it is determined that no search request has been received, the process goes to end and waits for a search request for similar image data from the server 10. On the other hand, if it is determined that a search request has been received from the server 10, the process of step S52 is executed.

[0089] <Step S52> In step S52, the image data having a feature amount similar to the feature amount of the transmitted similar image data is searched for in the database provided by the image data processing device. If image data having a similar feature amount is found in the process of searching, the process of step S53 is executed. For example, the image data having a feature amount similar to the feature amount of the vehicle CC in IMG-B of FIG. 4, the feature amount of the vehicle CB in IMG-C, and the feature amount of the road marking RM and the bicycle BY in IMG-D.

[0090] <Step S53> In step S53, if similar image data is searched for and found (YES judgement), the process goes to step S54, whereas if similar image data is not found (NO judgement), the process goes to step S55.

[0091] <Step S54> In step S54, the similar image data found as a result of the search is transmitted to the server 10. When the transmission of the similar image data is completed, the process goes to END and waits for the next search request.

[0092] <Step S55> In step S55, the result of the search, that similar image data has not been found, is transmitted to the server 10. When the transmission of the fact that similar image data has not been found is completed, the process goes to END and waits for the next search request.

[0093] When the server-side control unit of the server 10 receives similar image data sent from the vehicle-side control unit 11, it selects similar image data. Re-learning using all the similar image data sent would be too wasteful and inefficient. For this reason, similarity of feature quantities is used as a criterion for selecting similar image data.

[0094] Next, this control flow will be described with reference to Figures 10 to 13. Figure 10 shows a control flow for determining the similarity of feature amounts, and Figures 11 to 13 show a control flow for selecting image data having similar feature amounts.

[0095] First, a control flow for determining the similarity of feature quantities will be described with reference to FIG.

[0096] <Step S60> In step S60, an inference process is performed using an AI model on the similar image data from the vehicle-side control device 11. When the inference process is completed, the process of step S61 is executed.

[0097] <Step S61> In step S61, it is determined whether or not there is an area of ​​the same class as the target object to be collected in the inference processing result. If it is determined that there is no area of ​​the target object of the same class, the process returns to step S60 and performs inference processing of the next similar image data. On the other hand, if it is determined that there is an area of ​​the same class, the process of steps S62 to S63 is performed.

[0098] <Step S62> to <Step S63> In step S62, if it is determined that there is an area of ​​the same class as the target object to be collected, the feature amount of this area is calculated. Once the feature amount is calculated, the process of step S64 is executed. At this time, the process of step S63 is also executed in parallel. In step S63, the feature amount of the image data of the area to be collected is calculated and set by the designer or developer. Here, the feature amount is calculated based on the following concept.

[0099] In this embodiment, the similarity is determined from the difference between the average pixel value of the similar image data calculated in step S62 and the average pixel value of the image data in the collection target area. If this similarity is satisfied, the image data can be used as a candidate for re-learning.

[0100] Since image data can be expressed using three elements, R / G / B, the image data of the collection target image area in step S63 is expressed as Target_R(i), Target_G(i), and Target_B(i). Here, "i" is the i-th data in the image (i=0 to N-1), and "N" is the number of pixels in the image. As a result, the average value of the collection target image area can be calculated using the following "Equation 1".

[0101]

number

[0102] Similarly, similar image data are expressed as Input_R(i), Input_G(i), and Input_B(i). The average value of similar image data can also be calculated using the following "Equation 2." Here, assuming that the image area and image size of the collection target are different, the calculation is made with the number of pixels in the image = M.

[0103]

number

[0104] Once the average values ​​of the image data are calculated using "Equation 1" and "Equation 2", the process proceeds to step S64.

[0105] <Step S64> In step S64, the feature amounts (average values ​​of image data) calculated in steps S62 and S63 are compared to calculate the difference value (diff) between the two. The difference value (diff) is calculated using the following "Equation 3." This difference value (diff) represents the "similarity."

[0106]

number

[0107] Once the difference value (diff) is found, the process of step S65 is executed. The above-mentioned method is a method for calculating the similarity between the collection target image region and the input image region using the image average value, but it is also possible to calculate the similarity using a histogram or a correlation coefficient.

[0108] In addition, the above-mentioned method handles data in which the image data to be compared is composed of three R / G / B images, but the method can also handle data composed of 128 horizontal pixels × 96 vertical pixels × × 512 channels shown in the region candidate extraction unit in Figure 8.

[0109] <Step S65> In step S65, it is determined whether the difference value (diff) calculated in step S64 is smaller than a predetermined similarity threshold value. As can be seen from "Equation 3," the smaller the difference value (diff), the greater the similarity. Therefore, if it is determined that the difference value (diff) is smaller than the similarity threshold value (YES determination), the process of step S66 is executed. On the other hand, if it is determined that the difference value (diff) is greater than the similarity threshold value (NO determination), the process returns to step S60 and similar processing steps are executed for other similar image data.

[0110] <Step S66> In step S66, since the difference value (diff) of the feature amount of the similar image data satisfies the given similarity threshold, the image data is deemed valid for re-learning and is notified to the AI ​​learning block.

[0111] The similar image data that is deemed to be next valid is selected by image selection unit 15A for re-learning. The selection conditions are the same as those shown in Figures 5 to 7. The selection process flow will be described below.

[0112] 11 is a flow chart showing a process for determining the inference result of a target object. Each process step will be explained below.

[0113] <Step S70> In step S70, similar image data is input from the similar image data collection unit 15 A. Once the image data is input, the process of step S71 is executed.

[0114] <Step S71> In step S71, target inference processing is performed based on the input similar image data. This inference infers the "class information" of the target and the "position information" of the target. The "class information" is that the target is a vehicle, motorcycle, pedestrian, etc., and the "position information" is the coordinate information of the area surrounding the target. After the target inference processing is performed, the process of step S72 is performed.

[0115] <Step S72> In step S72, the inference result of the target inferred in step S71 is output. This output of the inference result is executed for all targets in the similar image data. When the inference result is output, the process of step S74 is executed. However, the process of step S73 is also executed in parallel.

[0116] <Step S73> In step S73, a designer or developer generates an inference result correct value for each target. The inference result correct value is the class information of the target and the position information of the target, and corresponds to the "class information" and "position information of the target" inferred in step S71. The generated inference result correct value is compared with the class information of the target and the position information of the target inferred in step S74.

[0117] <Step S74> ~ <Step S75> In step S74, the inference result of the target of the image data output in step S72 is compared with the correct inference result value of the target generated in step S73.

[0118] As a result of the comparison, if the inference result and the correct inference result value match (NO judgement), the process goes to END, and if they do not match (YES judgement), the process of step S76 is executed.

[0119] <Step S76> In step S76, the similar image data is selected as image data for re-learning. The selected similar image data is sent to the AI ​​learning unit 15B, where re-learning is performed.

[0120] 12 is a flow chart showing a process for determining the reliability of a target. Each process step will be explained below.

[0121] <Step S80> In step S80, similar image data is input from the similar image data collection unit 15 A. Once the image data is input, the process of step S81 is executed.

[0122] <Step S81> In step S81, target inference processing is performed based on the input similar image data. This inference infers the "reliability" of the target. Here, the "reliability" is numerical information indicating the degree of confidence with which the inference was made. After the target inference processing is performed, the process of step S82 is performed.

[0123] <Step S82> In step S82, the reliability of the target inferred in step S81 is output. This reliability output is performed for all targets in the image data. After the reliability is output, the process of step S84 is executed. However, the process of step S83 is also executed in parallel.

[0124] <Step S83> In step S83, a reliability threshold for each target is set by a designer or developer. Here, the reliability threshold is set to a lower limit of the reliability for recognizing a target included in image data. The set reliability threshold is compared with the reliability of the target output in step S82.

[0125] <Step S84> ~ <Step S85> In step S84, the reliability of the target in the image data output in step S82 is compared with the target reliability threshold value generated in step S83.

[0126] As a result of the comparison, if the reliability in step S82 exceeds the reliability threshold (NO judgment), the process goes to END, and if it is below the reliability threshold (YES judgment), the process of step S86 is executed.

[0127] <Step S86> In step S86, the similar image data is selected as image data for re-learning. The selected similar image data is sent to the AI ​​learning unit 15B, where re-learning is performed.

[0128] 13 is a flow chart showing a process for determining an area to be noted for recognizing a target (a determination basis area). Each process step will be explained below.

[0129] <Step S90> In step S90, similar image data is input from the similar image data collection unit 15 A. Once the image data is input, the process of step S91 is executed.

[0130] <Step S91> In step S91, target inference processing is performed based on the input image data. This inference infers the "judgment basis area" of the target. Here, the "judgment basis area" is information on an area to be focused on in order to recognize the target. After the target inference processing is performed, the processing of step S92 is performed.

[0131] <Step S92> In step S92, the judgment grounds area of ​​the target inferred in step S91 is output. This judgment grounds area is the area that was focused on when the target was inferred by inference. This output of the judgment grounds area is performed for all targets in the image data. Once the judgment grounds area is output, the process of step S94 is executed. However, the process of step S93 is also executed in parallel.

[0132] <Step S93> In step S93, a designer or developer sets a judgment basis area for each target. Here, the judgment basis area is set for all targets included in the image data. The set judgment basis area is compared with the judgment basis area of ​​the target output in step S92.

[0133] <Step S94> ~ <Step S95> In step S94, the determination basis region of the target in the image data output in step S92 is compared with the determination basis region of the target set in step S93.

[0134] As a result of the comparison, if the determination grounds region in step S92 matches the set determination grounds region (NO determination), the process goes to END, and if they do not match (YES determination), the process of step S96 is executed.

[0135] <Step S96> In step S96, the similar image data is selected as image data for re-learning. The selected similar image data is sent to the AI ​​learning unit 15B, where re-learning is performed.

[0136] When the selection of image data for re-learning according to FIGS. 11 to 13 is completed, re-learning is executed in step S16 of FIG. 3, and when this is completed, it is verified whether or not the inference result has been improved in step S17.

[0137] Here, whether or not the inference result in step S17 has improved is determined by re-executing the processing flow of Figures 11 to 13, and determining that there has been improvement if the inference result after learning matches a separately specified inference result output correct value, or if the reliability after learning exceeds a separately specified reliability threshold, or if the judgment basis area matches a judgment basis area correct value.

[0138] With the above-described configuration, for input data for which AI inference processing has not been performed correctly, features of the input data can be extracted, and similar data similar to these features can be collected from elsewhere and re-learned, making it possible to properly collect the required image data. EXAMPLES

[0139] Next, a second embodiment of the present invention will be described. In the above-described embodiment, the AI ​​inference block 12, the AI ​​verification block 13, the learning data collection block 14, and the AI ​​learning block 15 are provided in the server-side control unit of the server 10.

[0140] In contrast, in the second embodiment, the AI ​​inference block 12 and the learning data candidate collection block 14 are provided in the vehicle-side control device 11, and the AI ​​verification block 13 and the AI ​​learning block 15 are provided in the server-side control unit. The AI ​​inference block 12 includes an AI inference result correctness determination unit 12D that determines whether the inference result is correct. As a result, image data with an incorrect inference is input to the AI ​​verification unit 13A.

[0141] Here, the AI ​​inference block 12 and learning data candidate collection block 14 provided in the vehicle-side control device 11, and the AI ​​verification block 13 and AI learning block 15 provided in the server-side control unit of the server 10, perform the same functions and operations as in the first embodiment.

[0142] The present invention having the above-mentioned configuration is an AI model learning system for re-learning an AI model, characterized in that it has an AI inference processing unit that performs inference processing using an AI model on input data, an AI verification unit that acquires the input data as consideration data when the inference processing in the AI ​​inference processing unit is not performed correctly, a feature extraction unit that extracts features in the consideration data, a similar data collection unit that collects similar data that is similar to the features, and an AI learning unit that re-learns the AI ​​model using the similar data collected by the similar data collection unit.

[0143] According to this, for input data for which AI inference processing has not been performed correctly, features of the input data are extracted, and similar data that resembles these features is collected from elsewhere and re-learned, making it possible to properly collect the required image data.

[0144] The present invention is not limited to the above-described embodiments, but includes various modified examples. The above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace other configurations with respect to the configuration of each embodiment. [Explanation of symbols]

[0145] 10...server, 11...vehicle-side control device, 12...AI inference block, 12A...AI inference processing unit, 12B...AI inference defect candidate detection unit, 12C...input data acquisition unit, 13...AI verification block, 13A...AI verification unit, 13B...feature collection unit, 14...learning data candidate collection block, 14A...similar image data collection unit, 15...AI learning block, 15A...image selection unit, 15B...AI learning unit, 15C...AI update unit.

Claims

1. An AI model learning system for re-learning an AI model, An AI inference processing unit that performs inference processing using the AI ​​model on input data; An AI verification unit that acquires the input data as consideration data when the inference processing in the AI ​​inference processing unit is not performed correctly; A feature extraction unit that extracts features from the study data; a learning data candidate collection unit that collects similar data similar to the feature quantity; An AI learning unit that re-learns the AI ​​model using similar data collected by the learning data candidate collection unit; An AI model learning system comprising:

2. The AI ​​model learning system according to claim 1, When an inference result obtained by inputting the input data into the AI ​​model differs from a separately specified correct inference result value, the AI ​​verification unit sets a region of the input data as a feature extraction region. An AI model learning system characterized by:

3. The AI ​​model learning system according to claim 1, When the reliability of the input data when the input data is input to the AI ​​model is below a separately specified reliability threshold, the AI ​​verification unit sets a region of the input data as a feature extraction region. An AI model learning system characterized by:

4. The AI ​​model learning system according to claim 1, When a judgment basis region when the input data is input to the AI ​​model is different from a separately specified judgment basis region correct answer value, the AI ​​verification unit sets the judgment basis region as a feature extraction region. An AI model learning system characterized by:

5. The AI ​​model learning system according to any one of claims 2 to 4, The feature extraction unit extracts the feature using the input data corresponding to the feature extraction region. An AI model learning system characterized by:

6. The AI ​​model learning system according to any one of claims 2 to 4, The feature extraction unit extracts the feature using intermediate data obtained when the input data corresponding to the feature extraction region is input to the AI ​​model. An AI model learning system characterized by:

7. The AI ​​model learning system according to claim 5, The learning data candidate collection unit outputs the similar data to an AI learning unit when there is an area in which the similarity to the feature exceeds a separately specified similarity threshold. An AI model learning system characterized by:

8. The AI ​​model learning system according to claim 7, The AI ​​learning unit is When the similar data is input to the AI ​​model, if the inference result differs from the inference result correct value, or if the reliability falls below the reliability threshold, or if the judgment basis area differs from the judgment basis area correct value, the similar data is selected as learning data. An AI model learning system characterized by:

9. An AI model training method for re-training an AI model, comprising: An AI inference process that performs AI inference on input data; An AI verification process for verifying an inference result of the AI ​​inference; an improvement necessity determination process for determining whether or not the inference result needs improvement; A feature extraction process for extracting features from the input data when it is determined that the improvement is necessary; A similar data collection process for collecting similar data similar to the extracted feature amount; and executing an AI learning process for re-learning using the similar data. An AI model learning method comprising:

10. The AI ​​model learning method according to claim 9, After the re-learning by the AI ​​learning process, an improvement determination process is executed to determine whether the AI ​​inference has been improved; The improvement determination process includes: If the inference result after the re-learning matches a separately specified inference result output correct answer value, or if the reliability after learning exceeds a separately specified reliability threshold value, or if the judgment grounds area matches the judgment grounds area correct answer value, it is determined that there has been an improvement. An AI model learning method comprising:

11. An AI model learning system for re-learning an AI model for autonomously driving a vehicle, An AI inference function unit that performs inference processing using the AI ​​model on image data sent from the vehicle; an AI verification function unit that sets a target in the image data as a feature acquisition area when the inference process in the AI ​​inference function unit is not performed correctly; a feature extraction function unit that extracts a feature of the image data of the feature acquisition region; a similar image data collection function unit that requests a plurality of other vehicles to collect similar image data similar to the feature amount, and collects the similar image data from the other vehicles; An AI learning function unit that re-learns the AI ​​model using the similar image data collected by the similar image data collection function unit; An AI model learning system comprising:

12. The AI ​​model learning system according to claim 11, The AI ​​inference function unit includes an AI inference processing unit and an AI inference defect candidate detection unit, The AI ​​inference processing unit infers candidates for image data that are likely to be targets from the input image data, and the image data is sent to the AI ​​inference defect candidate detection unit, and the AI ​​inference defect candidate detection unit detects image data for which the inference process may not have been performed correctly and inputs the image data to the AI ​​verification function unit, The AI ​​verification function unit includes an AI verification unit and a feature extraction unit, The AI ​​verification unit judges whether the target of the image data for which inference processing may not have been performed correctly is different from a correct answer value, whether the target satisfies a threshold value, etc., and when it is judged that the target is different from the correct answer value or does not satisfy the threshold value, the AI ​​verification unit sets the area in which the target exists as a feature extraction area, The feature extraction unit extracts features of the target from the image data of the set feature extraction region and inputs the features to the similar image data collection function unit; The similar image data collection function unit wirelessly transmits the feature from the feature extraction unit to vehicle-side control devices of the other vehicles equipped with automatic driving devices, and the vehicle-side control device searches for similar image data having the feature similar to the transmitted feature from the image data stored in its own storage means and wirelessly transmits the searched image data back to the similar image data collection function unit, and the similar image data collection function unit inputs the received similar image data to the AI ​​learning function unit, The AI ​​learning function unit includes an image selection unit, an AI learning unit, and an AI update unit, The image selection unit selects the similar image data sent from the similar image data collection function unit as learning data and inputs it to the AI ​​learning unit, The AI ​​learning unit re-learns the AI ​​model using the selected similar image data, and the AI ​​update unit updates the AI ​​model. An AI model learning system characterized by:

13. The AI ​​model learning system according to claim 12, The vehicle-side control device is provided with an image data storage function unit that stores the image data to be input to the AI ​​verification function unit, on the condition that the user's consent is obtained in advance, and when the AI ​​model of the vehicle-side control device is updated by the AI ​​model re-learned by the AI ​​learning function unit, a confirmation function unit is provided that can confirm the improvement effect of the updated AI model on the stored image data. An AI model learning system characterized by:

14. The AI ​​model learning system according to claim 12, The AI ​​inference function unit and the similar image data collection function unit are mounted on the vehicle-side control device, and the AI ​​verification function unit and the AI ​​learning function unit are mounted on a server-side control unit of a server. An AI model learning system characterized by:

Images