Ai model training system and ai model training method

Abstract

The present invention comprises: an AI inference processing unit (12A) that performs inference processing on input data in an AI model; an AI verification unit (13A) that acquires, as examination data, input data when the AI inference processing unit (12A) has not correctly performed the inference processing; a feature amount extraction unit (13B) that extracts a feature amount in the examination data; a similar data collection unit (14A) that collects similar data similar to the feature amount; and an AI training unit (15B) that uses the similar data collected by the similar data collection unit (14A) to perform retraining on the AI model. In other words, according to the present invention, it is possible to appropriately collect required recognition data by extracting a feature amount of input data from input data on which AI inference processing has not been performed correctly, and collecting similar data from other data to perform the retraining.

Description

AI model learning system and AI model learning method

[0001] The present invention relates to an AI model learning system and an AI model learning method for an automatic driving device that automatically drives a vehicle such as an automobile.

[0002] Recently, development of autonomous driving, which allows automobiles to drive autonomously, has progressed. To achieve this autonomous driving, it is effective to use AI functions to control the vehicle's driving. To achieve high inference accuracy in the AI ​​model installed in the automobile, it is important to verify and analyze the AI ​​model, particularly the recognition model, using a sufficient amount of recognition data. Recognition data is typically image data, and the following explanation will be based on image data.

[0003] In order to ensure driving safety, the verification and analysis of this AI model requires the comprehensive collection of image data under a variety of actual driving conditions, spending an enormous amount of time. In particular, to enable autonomous driving of automobiles, image data of the driving environment under a variety of driving conditions is necessary. For this reason, image data is collected while the automobile is actually driving, and then verified and analyzed.

[0004] However, when collecting image data, a large amount of image data of scenes that are easy to photograph is generally collected, and image data of unusual scenes is not collected. As a result, various image data are not collected comprehensively, which makes it difficult to thoroughly verify and analyze the data.

[0005] To resolve this situation, a technique for comprehensively collecting image data is known, for example, as described in Patent Document 1 (WO 2022 / 107595).

[0006] Patent Document 1 discloses that, for the purpose of efficiently relearning a recognition model, the system includes a collection timing control unit that controls the timing of collecting image data candidates that are candidates for image data to be used in relearning the recognition model, and an image data collection unit that selects image data from the collected image data candidates based on at least one of the characteristics of the image data candidates and the similarity with stored image data.

[0007] The recognition model is used to recognize a predetermined recognition target (hereinafter, sometimes referred to as a target) around the vehicle, and the image data collection unit selects appropriate image data from image data candidates including image data obtained by photographing the surroundings of the vehicle with an imaging means such as a camera installed in the vehicle.

[0008] International Publication No. 2022 / 107595

[0009] Incidentally, the image data collection method described in Patent Document 1 determines whether or not to collect image data based on a certain external event (for example, an event such as the occurrence of construction work, the occurrence of an accident, rainfall conditions, or snowfall conditions), and classifies the image data collected based on this external event according to the output result (reliability) of the recognition model.

[0010] However, with this image data collection method, image data cannot be collected unless a relevant external event is encountered. Furthermore, this image data collection method uniformly collects unnecessary image data, resulting in unnecessary verification and analysis. Given these circumstances, there is a need to appropriately collect the necessary image data.

[0011] An object of the present invention is to provide an AI model learning system and an AI model learning method that can appropriately collect the image data required for relearning to verify and analyze an AI model.

[0012] The present invention is an AI model learning system for relearning an AI model, characterized by having an AI inference processing unit that performs inference processing using an AI model on input data, an AI verification unit that acquires input data as consideration data when the inference processing in the AI ​​inference processing unit is not performed correctly, a feature extraction unit that extracts features in the consideration data, a similar data collection unit that collects similar data that is similar to the features, and an AI learning unit that re-learns the AI ​​model using the similar data collected by the similar data collection unit.

[0013] According to the present invention, for input data for which AI inference processing has not been performed correctly, feature quantities of the input data are extracted, and similar data that resembles these feature quantities is collected from elsewhere and re-learned, thereby making it possible to properly collect the required image data.

[0014] FIG. 1 is an explanatory diagram illustrating the concept of an AI model learning system according to the present invention. FIG. 2 is a block diagram illustrating the configuration of an AI model learning system according to a first embodiment of the present invention. FIG. 3 is a flowchart illustrating a control flow of the AI ​​model learning system according to the first embodiment of the present invention. FIG. 4 is an explanatory diagram illustrating an example of a recognition error when recognizing a target from image data using an AI model. FIG. 5 is a flowchart illustrating a first control flow for cutting out a region of a target recognition error shown in FIG. 4. FIG. 6 is a flowchart illustrating a second control flow for cutting out a region of a target recognition error shown in FIG. 4. FIG. 7 is a flowchart illustrating a third control flow for cutting out a region of a target recognition error shown in FIG. 4. FIG. 8 is an explanatory diagram illustrating an example of extracting features from a feature extraction unit using a neural network. FIG. 9 is a flowchart illustrating a control flow when searching for similar image data in a vehicle-side control device. FIG. 10 is a control flowchart when determining the validity of similar image data from the vehicle-side control device. FIG. 11 is a control flowchart illustrating a first control flow for selecting similar image data to be used for re-learning. FIG. 12 is a control flowchart illustrating a second control flow for selecting similar image data to be used for re-learning. FIG. 13 is a block diagram illustrating the configuration of an AI model learning system according to a second embodiment of the present invention.

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiment, and various modifications and application examples within the technical concept of the present invention are also included within its scope.

[0016] 1 shows the configuration of an AI learning system to explain the concept of a first embodiment of the present invention. This embodiment is composed of a server 10 equipped with a server-side control unit having an AI function unit, a vehicle (here, an automobile) equipped with a vehicle-side control device 11 also having an AI function unit, and communication means (not shown) provided in each of the server 10 and the vehicle so that they can communicate wirelessly.

[0017] Furthermore, as described below, the vehicle-side control device 11 has the function of searching for similar data (including matches) to input data that cannot be inferred by the AI ​​model at the request of the server 10 and providing the data to the server 10.

[0018] Here, the AI ​​function unit of the server 10 is equipped with an AI model (here, a recognition model) to be verified. The AI ​​function unit of the vehicle-side control device is a current AI function unit capable of autonomously driving the vehicle. The recognition data here is image data used to recognize predetermined recognition targets (targets, etc.) around the vehicle. The recognition targets are targets such as other vehicles, pedestrians, bicycles, and traffic signs around the vehicle, which are required for autonomous driving.

[0019] Now, the vehicle-side control device 11 transmits image data that may have been recognized incorrectly by its own AI model to the server 10 via communication means. Note that this image data is positioned as a candidate for recognition error. The server-side control unit of the server 10 has at least two functions: an "evaluation / verification function unit" and a "relearning function unit."

[0020] The "Evaluation and Verification Function Unit" is equipped with model analysis and model visualization functions, and is configured to allow developers and designers to verify the analysis results on screen. In addition, model analysis performs verification using an AI model on the received image data, and extracts (area extraction) scenes where the AI ​​model cannot correctly infer targets in the image data.

[0021] For example, image 10A shows a deviation in the gaze area. Images (IMG.1 to IMG.3) show the deviation of the inferred value (shown by the solid line) from the correct value (shown by the dashed line) of the gaze area, with image (IMG.1) showing a small deviation and a certainty of 0.92. Image (IMG.2) showing a large deviation and a certainty of 0.51, and image (IMG.3) showing a medium deviation and a certainty of 0.76. For this reason, image (IMG.2) is considered to have an incorrectly inferred gaze area.

[0022] Then, feature quantities of the scene that cannot be correctly inferred are extracted, and data of these feature quantities is transmitted to the vehicle-side control devices 11 of many other vehicles. Here, there are no geographical restrictions on the other vehicle-side control devices 11, and the data is transmitted to vehicles located within the same country, for example. This makes it possible to expand the range in which image data similar to the feature quantities of the scene that cannot be correctly inferred exists.

[0023] On the other hand, many other vehicle-side control devices 11 search for image data (hereinafter referred to as similar image data) that is similar to the features sent from the features of the image data they hold, and if similar image data exists, they transmit the similar image data again to the server-side control unit of the server 10 via communication means.

[0024] The "relearning function unit" of the server-side control unit executes the relearning function using the received similar image data. The "relearning function unit" uses the collected similar image data to infer data using an AI model, and if it satisfies the training information, it confirms it as training data. Here, the training information is provided by developers and designers. Once relearning is complete, an update of the AI ​​model is executed for the vehicle-side control unit 11. The AI ​​model is updated as needed, and can also be reflected in the next autonomous driving device as an upgraded version of the AI ​​model.

[0025] Here, the "evaluation and verification function unit" and the "relearning function unit" may be provided in the same server 10, or may be provided separately in different servers 10. For example, the "evaluation and verification function unit" may be provided in the server 10 of a parts manufacturer that manufactures the vehicle-side control device 11, and the "relearning function unit" may be provided in a vehicle manufacturer that manufactures the vehicle.

[0026] In this way, for image data where AI inference processing has not been performed correctly, the features of the image data are extracted, and similar image data that is similar to these features is collected from other vehicles and re-learned, thereby making it possible to properly collect the required image data.

[0027] Next, the specific configuration and operation of the concept of this embodiment shown in Fig. 1 will be described below with reference to the drawings. Note that in the following description, the "evaluation and verification function unit" and the "relearning function unit" are described as being provided in the same server 10.

[0028] 2 shows the control block of the AI ​​model learning system, which is constructed in the server-side control unit of the server 10. The control block is composed of hardware such as a computer and logic circuits, and software that executes computer operations. For this reason, the control block can be considered a functional block.

[0029] 2, the AI ​​model learning system of this embodiment is characterized by having an AI inference block 12 that performs inference processing using a recognition model on input image data, an AI verification block 13 that verifies the correctness of the inference processing using this recognition model, a learning data candidate collection block 14 that collects learning data candidates that are similar image data, and an AI learning block 15 that re-learns the AI ​​using the learning candidate data collected by the learning data candidate collection block 14. Note that "block" can also be read as "functional unit."

[0030] The AI ​​inference block 12 includes at least an AI inference processing unit 12A, an AI inference defect candidate detection unit 12B, and an input data acquisition unit 12C.

[0031] The AI ​​inference processing unit 12A, for example, infers candidate image data that is likely to be a target from the input image data. This image data is sent to the AI ​​inference defect candidate detection unit 12B, which detects image data for which the inference process may not have been performed correctly. The input data acquisition unit 12C acquires image data for which the inference process may not have been performed correctly and inputs it to the AI ​​verification block 13.

[0032] The AI ​​verification block 13 includes at least an AI verification unit 13A and a feature extraction unit 13B.

[0033] The AI ​​verification unit 13A determines whether a target object in image data for which inference processing may have been performed incorrectly differs from a correct value, satisfies a threshold, etc. Here, the correct value and threshold are set by a developer or designer. If it is determined that the target object differs from a correct value or does not satisfy a threshold, the area in which the target object exists is set as a feature extraction area.

[0034] Next, the feature extraction unit 13B extracts feature amounts of the target from the image data of the set feature extraction region. Note that the feature extraction is similar to a well-known method in image processing. The feature extraction unit 13B inputs the extracted feature amount data to the learning data candidate collection block 14.

[0035] The training data candidate collection block 14 includes at least a similar image data collection unit 14A.

[0036] The similar image data collection unit 14A wirelessly transmits (e.g., by push transmission or scheduled transmission) at least the feature data from the feature extraction unit 13B and search request data for similar image data having features similar to the feature data to other vehicles equipped with an automatic driving device. Here, the other vehicles are not geographically restricted and are vehicles located in a wide area (e.g., city unit, prefecture unit, or the entire country of Japan).

[0037] This increases the likelihood that similar image data exists, and also increases responsiveness because similar image data can be obtained from a database stored in each vehicle.

[0038] The vehicle-side control device 11 installed in the vehicle searches for image data stored in its own storage means (database) that has features similar to the feature data sent, and if matching image data is found, wirelessly transmits a reply to the similar image data collection unit 14A. If matching image data is not found, the vehicle-side control device 11 wirelessly transmits reply data to that effect to the similar image data collection unit 14A. There may be one similar image data or multiple similar image data. The similar image data collection unit 14A transmits the collected similar image data to the AI ​​learning block 15 as learning image data.

[0039] The AI ​​learning block 15 includes at least an image selection unit 15A, an AI learning unit 15B, and an AI update unit 15C.

[0040] The image selection unit 15A selects similar image data sent from the similar image data collection unit 14A as learning data. This selection is performed by determining whether the similar image data differs from a correct answer value and whether it satisfies a threshold value. Here, the correct answer value and threshold value are set by a developer or designer. If it is determined that the similar image data differs from a correct answer value or does not satisfy the threshold value, the similar image data is sent to the AI ​​learning unit 15B as learning data.

[0041] The AI ​​learning unit 15B performs well-known relearning using the selected similar image data. The AI ​​update unit 15C then updates the AI ​​model (here, the recognition model). The updated AI model is fed back to the AI ​​inference processing unit 12A of the AI ​​inference block 12, and the AI ​​model in the AI ​​inference processing unit 12A is updated.

[0042] In addition, when image data is input from the AI ​​inference block 12 to the AI ​​verification block 13, the vehicle-side control device 11 is equipped with an image data storage function unit that stores the image data input to the AI ​​verification block 13 within the vehicle, provided that the user's consent is obtained in advance, and when the AI ​​model of the vehicle-side control device 11 is updated by an AI model re-learned in the AI ​​learning block 15, it is equipped with an improvement effect confirmation function unit that allows the user to confirm the improvement effect of the updated AI model on the image data stored in the vehicle.

[0043] The operation of the control block explained above is actually executed by the computer constituting the server-side control unit through calculation processing. Next, the control flow of the computer's re-learning of similar image data will be explained with reference to FIG.

[0044] <Step S10> In step S10, AI inference is performed. This inference infers candidate image data that is likely to be a target from the input image data. Then, image data for which the inference process may have been performed incorrectly is detected from this image data. If image data for which the inference process has not been performed correctly is detected, the processes of steps S11 to S12 are performed.

[0045] Steps S11 to S12: In steps S11 to S12, a determination is made as to whether the target object in the image data for which the inference process may have been performed incorrectly differs from the correct value and whether the target object satisfies a threshold value. Here, the correct value and the threshold value are set by a developer or a designer.

[0046] If it is determined that the result is different from the correct value or does not satisfy the threshold, it is determined that improvement is necessary, and the area in which the target exists is set as the feature extraction area. The AI ​​verification steps of steps S11 to S12 will be described in detail later.

[0047] Once the feature extraction region has been set, the process proceeds to step S13. If it is determined in step S12 that no improvement is required, the process ends and the next activation timing is awaited.

[0048] <Step S13> In step S13, feature amounts of targets for which inference processing may have been performed incorrectly are extracted from the image data of the feature amount extraction region set in step S12. This feature amount extraction is similar to a well-known method in image processing. Once the feature amounts have been extracted, the process of step S14 is executed.

[0049] <Step S14> In step S14, the vehicle-side control devices 11 of many other vehicles are requested to collect similar image data. This request is executed by wirelessly transmitting (e.g., push transmission or scheduled transmission) the feature data obtained in step S13 and search request data for similar image data having features similar to the feature data to other vehicles equipped with an automatic driving device. Note that the search process of the vehicle-side control device 11 will be described in detail later.

[0050] After the request in step S14, the process of step S15 is executed when similar image data is returned from the vehicle-side control device 11. Here, since similar image data is expected to be returned from many other vehicle-side control devices 11, the received similar image data may be stored and step S15 may be executed at a predetermined time.

[0051] <Step S15> In step S15, the collected similar image data is analyzed to determine whether it is appropriate image data, and appropriate similar image data is selected. This selection is performed by determining whether the similar image data differs from the correct answer value and whether it satisfies a threshold value. The correct answer value and threshold value are set by the developer or designer. If it is determined that the data differs from the correct answer value or does not satisfy the threshold value, it is treated as learning data. The similar image data selection process in step S15 will be described in detail later. Once the similar image data is selected, the processes of steps S16 to S17 are executed.

[0052] <Steps S16> to <Steps S17> In step S16, the selected similar image data is used to perform well-known re-learning. When performing re-learning, the target recognition result is re-verified in step S17. If the re-verification shows that the recognition was not correct and the inference result has not been improved, the process returns to step S14 and similar processing steps are performed again. On the other hand, if the re-verification shows that the recognition was correct and the inference result has been improved, the process exits and waits for the next activation timing.

[0053] The above is the configuration of the AI ​​model learning system in this embodiment. Next, we will explain the main processing functions in the above-mentioned control flow.

[0054] First, a specific determination method for the verification steps S11 and S12 will be described with reference to Figures 4 to 7. Figure 4 visually shows the state in which input image data is subjected to inference processing and set in a feature extraction region, Figure 5 shows the process flow for determining the need for improvement based on the inference results, Figure 6 shows the process flow for determining the need for improvement based on the reliability, and Figure 7 shows the process flow for determining the need for improvement based on the determination basis.

[0055] 4, input image data IMG-A input at a certain time shows three vehicles CA, CB, and CC as targets toward the back of the screen. Similarly, a bicycle BY is present in the foreground of the screen. Also, a road marking RM indicating a right turn is depicted on the road.

[0056] When inference processing is performed using an AI model on such input image data, the image data IMG-B indicates the output state of the inference results for the target, and since vehicles CA, CB, and bicycle BY are recognized, the inference is correct, but there is a possibility that the inference for vehicle CC is incorrect. For this reason, new image data is needed to correctly infer vehicle CC.

[0057] Furthermore, when inference processing is performed using an AI model, image data IMG-C indicates the output state of reliability, and vehicle CA and bicycle BY have reliability levels exceeding a threshold (set to 0.50 in this case), so are correctly inferred. However, vehicle CB may not have been inferred correctly because its reliability does not exceed the threshold. Therefore, new image data is required to correctly infer vehicle CB. Vehicle CC has not been correctly inferred in the inference results, so it is not detected.

[0058] Furthermore, when inference processing is performed using an AI model, the image data IMG-D indicates the output state of the judgment basis for the target object, and the judgment basis for vehicles CA, CB, and CC corresponds to each vehicle, but for bicycle BY, the judgment basis is output for road marking RM, and there is a possibility that the judgment basis has not been inferred correctly. For this reason, new image data is needed to correctly infer the judgment basis for bicycle BY.

[0059] Next, the method for deriving each of the above-mentioned inference results will be explained using a control flow. Figures 5 to 7 show the process flow for determining the need for improvement based on the inference results.

[0060] 5 is a flow chart showing the process of determining the inference result of the target. Each process step will be explained below.

[0061] <Step S20> In step S20, image data for which there is a possibility that the inference process has not been performed correctly is input from the input data acquisition unit 12 C. Once the image data is input, the process of step S21 is executed.

[0062] <Step S21> In step S21, target inference processing is performed based on the input image data. This inference infers the "class information" of the target and the "position information" of the target. The "class information" is information indicating whether the target is a vehicle, motorcycle, pedestrian, etc., and the "position information" is coordinate information of the area surrounding the target. After the target inference processing is performed, the processing of step S22 is performed.

[0063] <Step S22> In step S22, the inference results of the targets inferred in step S21 are output. This output of the inference results is executed for all targets in the image data. Once the inference results are output, the process of step S24 is executed. However, the process of step S23 is also executed in parallel.

[0064] <Step S23> In step S23, a designer or developer generates an inference result correct value for each target. The inference result correct value is target class information and target position information, and corresponds to the target "class information" and "target position information" inferred in step S21. The generated inference result correct value is compared with the target class information and target position information inferred in step S24.

[0065] <Steps S24> to <Steps S25> In step S24, the inference result of the target of the image data output in step S22 is compared with the correct value of the inference result of the target generated in step S23.

[0066] As a result of the comparison, if the inference result and the correct inference result value match (NO judgment), the process goes to END, and if they do not match (YES judgment), the process goes to step S26. Returning to the image data IMG-B in Figure 4, vehicles CA, CB, and bicycle BY are objects whose inference result and the correct inference result value match, and vehicle CC is an object whose inference result and the correct inference result value do not match.

[0067] <Step S26> In step S26, the mismatched region of the target determined to be a mismatch is extracted. This mismatched region becomes the feature extraction region described above. As shown in image data IMG-B in FIG. 4, the vehicle CC is the target where the inference result and the correct value do not match. Then, feature extraction unit 13B extracts features from the image data of the extracted mismatched region. Once the mismatched region is extracted, the process ends.

[0068] 6 shows a process flow for determining the reliability of a target. Each process step will be explained below.

[0069] <Step S30> In step S30, image data for which there is a possibility that the inference process has not been performed correctly is input from the input data acquisition unit 12 C. Once the image data is input, the process of step S31 is executed.

[0070] <Step S31> In step S31, target inference processing is performed based on the input image data. This inference infers the "reliability" of the target. Here, "reliability" is numerical information indicating the degree of confidence with which the inference was made. After the target inference processing is performed, the processing of step S32 is performed.

[0071] <Step S32> In step S32, the reliability of the target inferred in step S31 is output. This reliability output is performed for all targets in the image data. Once the reliability is output, the process of step S34 is executed. However, the process of step S33 is also executed in parallel.

[0072] <Step S33> In step S33, a reliability threshold is set by a designer or developer. Here, the reliability threshold is set to the lower limit of the reliability when recognizing a target included in image data (here, set to 0.50). Note that the reliability threshold can also be set for each target. The set reliability threshold is compared with the reliability of the target output in step S32.

[0073] <Steps S34> to <Steps S35> In step S34, the reliability of the target in the image data output in step S32 is compared with the target reliability threshold generated in step S33.

[0074] As a result of the comparison, if the reliability in step S32 exceeds the reliability threshold (NO determination), the process goes to end, and if it is below the reliability threshold (YES determination), the process goes to step S36. Returning to IMG-C in Figure 4, vehicle CA and bicycle BY have reliability of 0.50 or more, so are targets whose reliability exceeds the reliability threshold, and vehicle CB has a reliability of 0.45, so is a target whose reliability is below the reliability threshold.

[0075] <Step S36> In step S36, a region of a target whose reliability is below the reliability threshold is extracted. This region becomes the feature extraction region described above. As shown in IMG-C in FIG. 4, vehicle CB is a target whose reliability is below the reliability threshold. Then, feature extraction unit 13B extracts features from the image data of the extracted region. When a region whose reliability is below the reliability threshold is extracted, the process ends.

[0076] 7 shows a flow of a process for determining a region to be noted (a region for determining a reason for a determination) for recognizing a target object. Each process step will be explained below.

[0077] <Step S40> In step S40, image data for which there is a possibility that the inference process has not been performed correctly is input from the input data acquisition unit 12 C. Once the image data is input, the process of step S41 is executed.

[0078] <Step S41> In step S41, target inference processing is performed based on the input image data. This inference infers the "judgment basis area" of the target. Here, the "judgment basis area" is information on the area to be focused on in order to recognize the target. After the target inference processing is performed, the processing of step S42 is performed.

[0079] <Step S42> In step S42, the judgment basis area of ​​the target inferred in step S41 is output. This judgment basis area is the area that was focused on when the target was inferred by inference. This output of the judgment basis area is performed for all targets in the image data. Once the judgment basis area is output, the process of step S44 is executed. However, the process of step S43 is also executed in parallel.

[0080] <Step S43> In step S43, a designer or developer sets a judgment basis area for each target. Here, the judgment basis area is set for all targets included in the image data. The set judgment basis area is compared with the judgment basis area of ​​the target output in step S42.

[0081] <Steps S44> to <Steps S45> In step S44, the determination basis area of ​​the target in the image data output in step S42 is compared with the determination basis area of ​​the target set in step S43.

[0082] As a result of the comparison, if the determination basis area in step S42 matches the set determination basis area (NO determination), the process goes to end, and if it does not match (YES determination), the process goes to step S46. Returning to the image data IMG-D in Figure 4, vehicles CA, CB, and CC are objects whose determination basis area in step S42 matches the set determination basis area, while bicycle BY is an object whose determination basis area in step S42 does not match the set determination basis area. In the case of bicycle BY, road marking RM is the determination basis area.

[0083] <Step S46> In step S46, the mismatched area of ​​the target determined to be a mismatch is extracted. This mismatched area becomes the feature extraction area described above. As shown in image data IMG-D in FIG. 4, the road sign RM and the bicycle BY are targets whose judgment basis areas do not match. Then, feature extraction unit 13B extracts features from the image data of the extracted mismatched area. Once the mismatched area is extracted, the process ends.

[0084] Then, the mismatched areas of the image data that need to be improved are designated as feature extraction areas, and feature amounts of the target are extracted. The feature amounts may be extracted using various methods. For example, the feature amounts are extracted using intermediate data obtained when the image data corresponding to the feature extraction area is input into an AI model. This state is shown in FIG. 8.

[0085] In Figure 8, the feature extraction unit of the AI ​​model is composed of a neural network consisting of a convolutional layer and a pooling layer, and features can be extracted by passing input image data through the convolutional layer and pooling layer in order. Note that the numbers for the image data input to the convolutional layer in Figure 8 indicate the "number of horizontal pixels x number of vertical pixels x number of channels." In this embodiment, intermediate data before being input to the class identification unit 20 in particular is extracted as features by the region candidate extraction unit 21.

[0086] Next, a method for collecting similar image data that resembles the extracted features will be described. When a request for collecting similar image data is sent from the server-side control unit to many other vehicles, the many other vehicles will receive this transmission request. Then, upon receiving this collection request, the vehicle-side control unit 11 of each vehicle searches for similar image data from the image data stored and held by itself. The processing flow of this data search will be described using Figure 9.

[0087] <Step S50> In step S50, the system waits for a search request for similar image data from the server 10. In this state, the process of step S51 is executed.

[0088] <Step S51> In step S51, it is determined whether or not a search request has been received from the server 10. If it is determined that a search request has not been received, the process exits and waits for a search request for similar image data from the server 10. On the other hand, if it is determined that a search request has been received from the server 10, the process of step S52 is executed.

[0089] <Step S52> In step S52, the database is searched for image data having features similar to those of the transmitted similar image data. If image data having similar features is found during this search process, the process of step S53 is executed. For example, this is image data having features similar to those of the vehicle CC of IMG-B in FIG. 4, the vehicle CB of IMG-C, and the road marking RM and bicycle BY of IMG-D.

[0090] <Step S53> In step S53, if similar image data is searched and found (YES determination), the process of step S54 is executed, whereas if similar image data is not found (NO determination), the process of step S55 is executed.

[0091] <Step S54> In step S54, the similar image data found as a result of the search is transmitted to the server 10. When the transmission of the similar image data is completed, the process exits to end and waits for the next search request.

[0092] <Step S55> In step S55, the result of the search, that similar image data has not been found, is transmitted to the server 10. When the transmission of the fact that similar image data has not been found is completed, the process exits to end and waits for the next search request.

[0093] The server-side control unit of the server 10 selects similar image data when it receives similar image data sent from the vehicle-side control unit 11. Re-learning using all of the similar image data sent would be wasteful and inefficient. For this reason, similar image data is selected using the similarity of feature values ​​as a criterion.

[0094] Next, this control flow will be explained using Figures 10 to 13. Figure 10 shows the control flow for determining the similarity of feature amounts, and Figures 11 to 13 show the control flow for selecting image data having similar feature amounts.

[0095] First, the control flow for determining the similarity of feature quantities will be described with reference to FIG.

[0096] <Step S60> In step S60, an AI model is used to perform inference processing on the similar image data from the vehicle-side control device 11. When the inference processing is completed, the processing of step S61 is executed.

[0097] <Step S61> In step S61, it is determined whether or not there is an area of ​​the same class as the target object to be collected in the results of the inference process. If it is determined that there is no area of ​​the target object of the same class, the process returns to step S60 and inference processing is performed for the next similar image data. On the other hand, if it is determined that there is an area of ​​the same class, the processes of steps S62 to S63 are performed.

[0098] <Steps S62> to <Steps S63> In step S62, if it is determined that there is an area of ​​the same class as the target object to be collected, the feature amount of this area is calculated. Once the feature amount is calculated, the process of step S64 is executed. At this time, the process of step S63 is also executed in parallel. In step S63, the feature amount of the image data of the area to be collected is calculated and set by the designer or developer. Here, the feature amount is calculated based on the following concept.

[0099] In this embodiment, the similarity is determined from the difference between the average pixel value of the similar image data calculated in step S62 and the average pixel value of the image data in the collection target area. If this similarity is satisfied, the image data can be used as a candidate for re-learning.

[0100] Since image data can be expressed using the three elements R / G / B, the image data of the image area to be collected in step S63 is expressed as Target_R(i), Target_G(i), and Target_B(i). Here, "i" is the i-th data in the image (i = 0 to N-1), and "N" is the number of pixels in the image. As a result, the average value of the image area to be collected can be calculated using the following "Equation 1."

[0101]

[0102] Similarly, similar image data are expressed as Input_R(i), Input_G(i), and Input_B(i). The average value of similar image data can also be calculated using the following "Equation 2." Here, assuming that the image area and image size of the collection target are different, the calculation is performed assuming the number of pixels in the image = M.

[0103]

[0104] Once the average values ​​of the image data are calculated using "Equation 1" and "Equation 2," the process proceeds to step S64.

[0105] <Step S64> In step S64, the feature amounts (average values ​​of image data) calculated in step S62 and step S63 are compared to calculate the difference value (diff) between them. The difference value (diff) is calculated using the following "Equation 3." This difference value (diff) represents the "similarity."

[0106]

[0107] Once the difference value (diff) is found, the process proceeds to step S65. The method described above is a method of calculating the similarity between the image region to be collected and the input image region using the image average value, but it is also possible to calculate the similarity using a histogram or a correlation coefficient.

[0108] Furthermore, the above-described method handles data in which the image data to be compared is made up of three R / G / B images, but it can also handle data made up of 128 horizontal pixels × 96 vertical pixels × 512 channels shown in the region candidate extraction unit of FIG. 8 in the same way.

[0109] <Step S65> In step S65, it is determined whether the difference value (diff) calculated in step S64 is smaller than a predetermined similarity threshold. As can be seen from "Equation 3," the smaller the difference value (diff), the more similar the images are. Therefore, if it is determined that the difference value (diff) is smaller than the similarity threshold (YES determination), the process of step S66 is executed. On the other hand, if it is determined that the difference value (diff) is greater than the similarity threshold (NO determination), the process returns to step S60, and similar processing steps are executed for other similar image data.

[0110] <Step S66> In step S66, the difference value (diff) of the feature amounts of the similar image data satisfies the given similarity threshold, so it is deemed valid as image data for re-learning and notified to the AI ​​learning block.

[0111] The next similar image data that is deemed valid is selected by the image selection unit 15A for re-learning. The selection conditions are the same as those shown in Figures 5 to 7. The selection process flow will be described below.

[0112] 11 is a flow chart showing the process of determining the inference result of the target. Each process step will be explained below.

[0113] <Step S70> In step S70, similar image data is input from the similar image data collection unit 14 A. Once the image data is input, the process of step S71 is executed.

[0114] <Step S71> In step S71, target inference processing is performed based on the input similar image data. This inference infers the "class information" and "position information" of the target. The "class information" indicates whether the target is a vehicle, motorcycle, pedestrian, etc., and the "position information" is coordinate information of the area surrounding the target. After the target inference processing is performed, the processing of step S72 is performed.

[0115] <Step S72> In step S72, the inference results of the targets inferred in step S71 are output. This output of the inference results is executed for all targets in the similar image data. Once the inference results are output, the process of step S74 is executed. However, the process of step S73 is also executed in parallel.

[0116] <Step S73> In step S73, a designer or developer generates an inference result correct value for each target. The inference result correct value is target class information and target position information, and corresponds to the target "class information" and "target position information" inferred in step S71. The generated inference result correct value is compared with the target class information and target position information inferred in step S74.

[0117] <Steps S74 to S75> In step S74, the inference result of the target of the image data output in step S72 is compared with the correct value of the inference result of the target generated in step S73.

[0118] As a result of the comparison, if the inference result and the correct inference result value match (NO judgment), the process goes to end, and if they do not match (YES judgment), the process of step S76 is executed.

[0119] <Step S76> In step S76, similar image data is selected as image data for re-learning. The selected similar image data is sent to the AI ​​learning unit 15B, where re-learning is performed.

[0120] 12 shows a process flow for determining the reliability of a target. Each process step will be explained below.

[0121] <Step S80> In step S80, similar image data is input from the similar image data collection unit 14 A. Once the image data is input, the process of step S81 is executed.

[0122] <Step S81> In step S81, target inference processing is performed based on the input similar image data. This inference infers the "reliability" of the target. Here, "reliability" is numerical information indicating the degree of confidence with which the inference was made. After the target inference processing is performed, processing in step S82 is performed.

[0123] <Step S82> In step S82, the reliability of the target inferred in step S81 is output. This reliability output is performed for all targets in the image data. Once the reliability is output, the process of step S84 is executed. However, the process of step S83 is also executed in parallel.

[0124] <Step S83> In step S83, a reliability threshold for each target is set by a designer or developer. Here, the reliability threshold is set to a lower limit of the reliability for recognizing a target included in image data. The set reliability threshold is compared with the reliability of the target output in step S82.

[0125] <Steps S84> to <Steps S85> In step S84, the reliability of the target in the image data output in step S82 is compared with the target reliability threshold value generated in step S83.

[0126] As a result of the comparison, if the reliability in step S82 exceeds the reliability threshold (NO determination), the process goes to end, and if it is below the reliability threshold (YES determination), the process of step S86 is executed.

[0127] <Step S86> In step S86, similar image data is selected as image data for re-learning. The selected similar image data is sent to the AI ​​learning unit 15B, where re-learning is performed.

[0128] 13 shows a flow of a process for determining a region to be noted (a region for determining a reason for a determination) for recognizing a target object. Each process step will be explained below.

[0129] <Step S90> In step S90, similar image data is input from the similar image data collection unit 14 A. Once the image data is input, the process of step S91 is executed.

[0130] <Step S91> In step S91, target inference processing is performed based on the input image data. This inference infers the "judgment basis area" of the target. Here, the "judgment basis area" is information on the area to be focused on in order to recognize the target. After the target inference processing is performed, the processing of step S92 is performed.

[0131] <Step S92> In step S92, the judgment basis area of ​​the target inferred in step S91 is output. This judgment basis area is the area that was focused on when the target was inferred by inference. This output of the judgment basis area is performed for all targets in the image data. Once the judgment basis area is output, the process of step S94 is executed. However, the process of step S93 is also executed in parallel.

[0132] <Step S93> In step S93, a designer or developer sets a judgment basis area for each target. Here, a judgment basis area is set for all targets included in the image data. The set judgment basis area is compared with the judgment basis area of ​​the target output in step S92.

[0133] <Steps S94> to <Steps S95> In step S94, the determination basis area of ​​the target in the image data output in step S92 is compared with the determination basis area of ​​the target set in step S93.

[0134] As a result of the comparison, if the determination basis area in step S92 matches the set determination basis area (NO determination), the process goes to end, and if they do not match (YES determination), the process of step S96 is executed.

[0135] <Step S96> In step S96, similar image data is selected as image data for re-learning. The selected similar image data is sent to the AI ​​learning unit 15B, where re-learning is performed.

[0136] When the selection of image data for re-learning according to FIGS. 11 to 13 is completed, re-learning is executed according to step S16 in FIG. 3, and when this is completed, it is verified whether or not the inference result has improved in step S17.

[0137] Here, whether or not the inference result in step S17 has improved is determined by re-executing the processing flow of Figures 11 to 13, if the inference result after learning matches a separately specified inference result output correct value, or if the reliability after learning exceeds a separately specified reliability threshold, or if the judgment basis area matches a judgment basis area correct value.

[0138] With the above configuration, for input data for which AI inference processing has not been performed correctly, the features of the input data can be extracted, and similar data that is similar to these features can be collected from elsewhere and re-learned, thereby making it possible to properly collect the required image data.

[0139] Next, a second embodiment of the present invention will be described. In the previously described embodiment, the AI ​​inference block 12, the AI ​​verification block 13, the learning data candidate collection block 14, and the AI ​​learning block 15 are provided in the server-side control unit of the server 10.

[0140] In contrast, in the second embodiment, the AI ​​inference block 12 and the learning data candidate collection block 14 are provided in the vehicle-side control device 11, and the AI ​​verification block 13 and the AI ​​learning block 15 are provided in the server-side control unit. The AI ​​inference block 12 includes an AI inference result correctness determination unit 12D that determines whether the inference result is correct. As a result, image data with an incorrect inference is input to the AI ​​verification unit 13A.

[0141] Here, the AI ​​inference block 12 and learning data candidate collection block 14 provided in the vehicle-side control device 11, and the AI ​​verification block 13 and AI learning block 15 provided in the server-side control unit of the server 10, perform the same functions and operations as in the first embodiment.

[0142] The present invention having the above-described configuration is an AI model learning system for relearning an AI model, characterized by having an AI inference processing unit that performs inference processing on input data using an AI model, an AI verification unit that acquires input data as consideration data when the inference processing in the AI ​​inference processing unit is not performed correctly, a feature extraction unit that extracts features in the consideration data, a similar data collection unit that collects similar data that is similar to the features, and an AI learning unit that re-learns the AI ​​model using the similar data collected by the similar data collection unit.

[0143] According to this, for input data for which AI inference processing has not been performed correctly, the features of the input data are extracted, and similar data that resembles these features is collected from elsewhere and re-learned, thereby making it possible to properly collect the required image data.

[0144] The present invention is not limited to the above-described embodiments, but includes various modifications. The above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace other configurations with respect to the configuration of each embodiment.

Claims

1. An AI model learning system for re-learning an AI model, comprising: an AI inference processing unit that performs inference processing on input data using the AI ​​model; an AI verification unit that acquires the input data as consideration data when the inference processing in the AI ​​inference processing unit is not performed correctly; a feature extraction unit that extracts features in the consideration data; a learning data candidate collection unit that collects similar data that is similar to the features; and an AI learning unit that re-learns the AI ​​model using the similar data collected by the learning data candidate collection unit.

2. An AI model learning system as described in claim 1, characterized in that, when the inference result when the input data is input to the AI ​​model differs from a separately specified correct inference result value, the AI ​​verification unit sets the area of ​​the input data as a feature extraction area.

3. An AI model learning system as described in claim 1, characterized in that, when the reliability of the input data when it is input to the AI ​​model falls below a separately specified reliability threshold, the AI ​​verification unit sets the area of ​​the input data as a feature extraction area.

4. An AI model learning system as described in claim 1, characterized in that, when the judgment basis area when the input data is input to the AI ​​model differs from a separately specified judgment basis area correct value, the AI ​​verification unit sets the judgment basis area as a feature extraction area.

5. An AI model learning system according to any one of claims 2 to 4, characterized in that the feature extraction unit extracts the features using the input data corresponding to the feature extraction region.

6. An AI model learning system according to any one of claims 2 to 4, characterized in that the feature extraction unit extracts the features using intermediate data obtained when the input data corresponding to the feature extraction region is input to the AI ​​model.

7. An AI model learning system as described in claim 5, characterized in that, when there is an area in which the similarity with the feature exceeds a separately specified similarity threshold, the learning data candidate collection unit outputs the similar data to the AI ​​learning unit.

8. An AI model learning system as described in claim 7, characterized in that the AI ​​learning unit selects the similar data as learning data when the inference result differs from the inference result correct value, or when the reliability is below the reliability threshold, or when the judgment basis area differs from the judgment basis area correct value, when the similar data is input to the AI ​​model.

9. An AI model learning method for re-learning an AI model, comprising: an AI inference process that performs AI inference on input data; an AI verification process that performs verification on the inference result of the AI ​​inference; an improvement necessity determination process that determines the need for improvement to the inference result; a feature extraction process that extracts features for the input data when it is determined that the improvement is necessary; a similar data collection process that collects similar data that is similar to the extracted features; and an AI learning process that performs re-learning using the similar data.

10. An AI model learning method as set forth in claim 9, wherein after the re-learning by the AI ​​learning process, an improvement determination process is executed to determine whether the AI ​​inference has been improved, and the improvement determination process determines that an improvement has occurred if the inference result after the re-learning matches a separately specified inference result output correct value, or if the reliability after learning exceeds a separately specified reliability threshold, or if the judgment basis area matches the judgment basis area correct value.

11. An AI model learning system for relearning an AI model for autonomously driving a vehicle, comprising: an AI inference function unit that performs inference processing using the AI ​​model on image data sent from the vehicle; an AI verification function unit that sets a target in the image data as a feature acquisition area when the inference processing by the AI ​​inference function unit is not performed correctly; a feature extraction function unit that extracts features of the image data in the feature acquisition area; a similar image data collection function unit that requests a plurality of other vehicles to collect similar image data similar to the features and collects the similar image data from the other vehicles; and an AI learning function unit that re-learns the AI ​​model using the similar image data collected by the similar image data collection function unit.

12. An AI model learning system according to claim 11, wherein the AI ​​inference function unit includes an AI inference processing unit and an AI inference defect candidate detection unit, the AI ​​inference processing unit infers candidates for the image data that are likely to be targets from the input image data, the image data is sent to the AI ​​inference defect candidate detection unit, and the AI ​​inference defect candidate detection unit detects the image data for which inference processing may not have been performed correctly and inputs it to the AI ​​verification function unit, the AI ​​verification function unit includes an AI verification unit and a feature extraction unit, the AI ​​verification unit judges whether the target of the image data for which inference processing may not have been performed correctly is different from a correct value, whether it satisfies a threshold value, etc., and when it is judged that the target is different from the correct value or does not satisfy the threshold value, the AI ​​verification unit sets the area in which the target exists as a feature extraction area, and the feature extraction unit extracts features of the target from the image data in the set feature extraction area and inputs them to the similar image data collection function unit, the similar image data collection function unit wirelessly transmits the features from the feature extraction unit to vehicle-side control devices of multiple other vehicles equipped with automatic driving devices, the vehicle-side control device searches for similar image data having features similar to the sent features from the image data stored in its own storage means and wirelessly transmits the searched result back to the similar image data collection function unit, the similar image data collection function unit inputs the received similar image data to the AI ​​learning function unit, the AI ​​learning function unit comprises an image selection unit, an AI learning unit, and an AI update unit, the image selection unit selects the similar image data sent from the similar image data collection function unit as learning data and inputs it to the AI ​​learning unit, the AI ​​learning unit re-learns the AI ​​model using the selected similar image data, and updates the AI ​​model in the AI ​​update unit.

13. An AI model learning system as described in claim 12, characterized in that the vehicle-side control device is provided with an image data storage function unit that stores the image data to be input to the AI ​​verification function unit, on the condition that prior consent of the user is obtained, and when the AI ​​model of the vehicle-side control device is updated by the AI ​​model re-learned by the AI ​​learning function unit, a confirmation function unit is provided that can confirm the improvement effect of the updated AI model on the stored image data.

14. An AI model learning system as described in claim 12, characterized in that the AI ​​inference function unit and the similar image data collection function unit are mounted on the vehicle-side control device, and the AI ​​verification function unit and the AI ​​learning function unit are mounted on a server-side control unit of a server.

Images