Information processing method and information processing system

Abstract

Provided is an information processing method capable of determining a training limit of a machine learning model. In the information processing method executed by a computer, evaluation data obtained is input to a first model to obtain a first inference result, and based on reference information of the evaluation data and the first inference result, an abnormality of the first inference result is determined; a second model is obtained through training of machine learning based on the determination result; the evaluation data is input to the second model to obtain a second inference result; based on the reference information of the evaluation data and the second inference result, an abnormality of the second inference result is determined; a third model is obtained through training of machine learning based on the determination result; the evaluation data is input to the third model to obtain a third inference result; based on the reference information of the evaluation data and the third inference result, an abnormality of the third inference result is determined; and in a case where the abnormality of the third inference result and the abnormality of the first inference result are identified as being the same, information related to a training limit of the first model is output.

Description

Technical Field

[0001] This disclosure relates to an information processing method executed by a computer and an information processing system that executes the information processing method. Background Technology

[0002] Deep learning-based object detection techniques that achieve high accuracy (e.g., refer to Non-Patent Document 1) are predicted to be practical for various applications.

[0003] In object detection technology that utilizes deep learning and other machine learning methods, as a countermeasure against detecting anomalies (including false detections and undetected objects), the following training data is used to train the model through machine learning, which is supplemented with data of the objects to be detected and processed where anomalies have occurred (see Patent Document 1).

[0004] (Existing technical literature)

[0005] (Patent Documents)

[0006] Patent Document 1: Japanese Patent Application Publication No. 2002-342739

[0007] (Non-patent literature)

[0008] Non-patent literature 1: Qijie Zhao, 6th author, "M2Det: A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network", November 2018, [online], arXiv, [retrieved February 26, 2020], Internet<URL∶https∶ / / arxiv.org / abs / 1811.04533>

[0009] However, even with additional training data, anomalies are sometimes not eliminated. Determining whether anomalies can be eliminated, i.e., whether the training has reached its limit, is difficult. Summary of the Invention

[0010] This disclosure provides information processing methods, etc., that can determine the training limits of machine learning.

[0011] One aspect of this disclosure relates to an information processing method executed by a computer. The method includes: obtaining evaluation data; inputting the evaluation data into a first model performing inference processing to obtain a first inference result; performing a first judgment based on reference information of the evaluation data and the first inference result, wherein the first judgment is a judgment of error or omission in the first inference result; obtaining a second model through first training, wherein the first training refers to machine learning training in training data containing the first evaluation data and one or both of the data similar to the first evaluation data, wherein the first evaluation data corresponds to the first inference result judged to have an error or omission; inputting the evaluation data into the second model to obtain a second inference result; and performing a second judgment based on reference information of the evaluation data and the second inference result, wherein the second judgment is... The process includes: judging errors or omissions in the second inference result; obtaining a third model through second training, whereby the second training refers to machine learning training in training data that includes the second evaluation data and one or both of similar data in the evaluation data, where the second evaluation data corresponds to the second inference result and has errors or omissions not present in the first inference result; inputting the evaluation data into the third model to obtain a third inference result; performing a third judgment based on the reference information of the evaluation data and the third inference result, whereby the third judgment is a judgment of errors or omissions in the third inference result; and outputting information related to the training limits of the first model when the errors or omissions in the third inference result are identified to be the same as those in the first inference result.

[0012] Furthermore, one solution disclosed herein involves an information processing system comprising: an inference processing unit; an anomaly detection unit; a model training unit; and a notification control unit. The inference processing unit obtains evaluation data, inputs the evaluation data into a first model performing inference processing, and obtains a first inference result. The anomaly detection unit performs a first judgment based on reference information of the evaluation data and the first inference result. The first judgment is a judgment of error or omission in the first inference result. The model training unit obtains a second model through first training. The first training refers to machine learning training in training data that includes the first evaluation data and one or both of the data similar to the first evaluation data. The first evaluation data corresponds to the evaluation data of the first inference result judged to have an error or omission. The inference processing unit further inputs the evaluation data into the second model to obtain a second inference result. The anomaly detection unit further performs a second judgment based on reference information of the evaluation data and the second inference result. The second judgment is a judgment on the error or omission of the second inference result. The model training unit further obtains a third model through second training. The second training refers to machine learning training in the training data by including one or both of the second evaluation data and data similar to the second evaluation data in the evaluation data. The second evaluation data is the evaluation data corresponding to the second inference result. The second inference result has errors or omissions that are not present in the first inference result. The inference processing unit further inputs the evaluation data into the third model to obtain a third inference result. The anomaly judgment unit further performs a third judgment based on the reference information of the evaluation data and the third inference result. The third judgment is a judgment on the error or omission of the third inference result. The notification control unit outputs information related to the training limit of the first model when it is identified that the error or omission of the third inference result is the same as the error or omission of the first inference result.

[0013] In addition, these general or specific solutions, besides the methods and systems described, can also be implemented by devices, integrated circuits, or computer-readable recording media such as CD-ROMs, or by any combination of devices, systems, integrated circuits, methods, computer programs, and recording media.

[0014] By utilizing the information processing methods disclosed herein, the training limits of machine learning can be determined. Attached Figure Description

[0015] Figure 1 This is a block diagram illustrating a functional configuration example of an information processing system that implements the information processing method according to the embodiment.

[0016] Figure 2 This is a flowchart illustrating an example of anomaly detection steps in an information processing method according to an embodiment.

[0017] Figure 3 This is a flowchart illustrating an example of the steps involved in an information processing method according to an embodiment. Detailed Implementation

[0018] (The insights that form the basis of this disclosure)

[0019] In previous technologies, inventors discovered the following problems.

[0020] For example, when object detection technology is used in applications requiring high reliability, such as autonomous driving, countermeasures against false detections and undetected anomalies are essential. This is because such anomalies could potentially lead to accidents resulting in injury or death.

[0021] In object detection techniques utilizing machine learning methods such as deep learning, the training of machine learning models, as a countermeasure against detected anomalies, typically uses training data, which includes data on objects that have experienced detected anomalies. However, even if the model obtained as a result of this training eliminates detected anomalies, the judgment regarding whether other detected anomalies occur within the model—in other words, whether the overall accuracy of object detection achieved through such training has improved or reached its limit—currently relies heavily on the experience or intuition of the person responsible for training the model. For example, to obtain a more accurate object detection model, an experienced person might explore changing the network structure, while a less experienced person might choose to look at the results of further training with additional training data. Thus, depending on the technical ability of the person performing the operation, there can be a problem of bias in judging the limits of machine learning training.

[0022] This disclosure addresses such problems and provides a solution to an information processing method executed by a computer. The method involves: obtaining evaluation data; inputting the evaluation data into a first model performing inference processing to obtain a first inference result; performing a first judgment based on reference information of the evaluation data and the first inference result, wherein the first judgment is a judgment of error or omission in the first inference result; obtaining a second model through first training, wherein the first training refers to machine learning training in training data containing the first evaluation data and one or both of the data similar to the first evaluation data, wherein the first evaluation data corresponds to the first inference result judged to have an error or omission; inputting the evaluation data into the second model to obtain a second inference result; and performing a second judgment based on reference information of the evaluation data and the second inference result. The second judgment is a judgment of errors or omissions in the second inference result; a third model is obtained through second training, the second training referring to machine learning training in the training data that includes the second evaluation data and one or both of the data similar to the second evaluation data in the evaluation data, the second evaluation data being the evaluation data corresponding to the second inference result, the second inference result having errors or omissions not present in the first inference result; the evaluation data is input into the third model to obtain a third inference result; a third judgment is performed based on the reference information of the evaluation data and the third inference result, the third judgment being a judgment of errors or omissions in the third inference result; and if it is identified that the errors or omissions in the third inference result are the same as those in the first inference result, information related to the training limit of the first model is output.

[0023] Therefore, the assessment of the limits of performance improvement of the inference model through machine learning training does not depend on the skill level of the person responsible for creating the inference model.

[0024] Alternatively, the evaluation data may be image data, the inference processing may be object detection processing, and the first inference result, the second inference result, and the third inference result may be object detection results.

[0025] That is, to improve the performance limit of the inference model without relying on the proficiency of the person who created the inference model, the inference model is a model for object detection processing of image data.

[0026] Alternatively, the object detection result may include a first detection box, which is a detection box of an object as a result of the object detection process, and the reference information may include a second detection box, which is a detection box of a reference object. The error or omission is determined based on the first detection box and the second detection box.

[0027] Anomalies in object detection can be judged as described above.

[0028] Alternatively, a notification can be executed by outputting the information indicating the possibility that the training limit has been reached, the notification being executed via an image or sound.

[0029] Therefore, in order to improve the performance of inference models, inference model creators are encouraged to consider methods other than supplementing training based on data that does not perform normal inference, thereby achieving operational efficiency.

[0030] Furthermore, one aspect of the information processing system disclosed herein includes: an inference processing unit; an anomaly detection unit; a model training unit; and a notification control unit. The inference processing unit obtains evaluation data, inputs the evaluation data into a first model performing inference processing, and obtains a first inference result. The anomaly detection unit performs a first judgment based on reference information from the evaluation data and the first inference result. The first judgment is a judgment of error or omission in the first inference result. The model training unit obtains a second model through first training, whereby the first training refers to machine learning training that includes the first evaluation data and one or both of similar data in training data. The first evaluation data corresponds to the evaluation data of the first inference result that is judged to have an error or omission. The inference processing unit further inputs the evaluation data into the second model to obtain a second inference result. The anomaly detection unit further performs a second judgment based on reference information from the evaluation data and the second inference result. The second judgment is a judgment on the error or omission of the second inference result. The model training unit further obtains a third model through second training. The second training refers to machine learning training in the training data by including one or both of the second evaluation data and data similar to the second evaluation data in the evaluation data. The second evaluation data is the evaluation data corresponding to the second inference result. The second inference result has errors or omissions that are not present in the first inference result. The inference processing unit further inputs the evaluation data into the third model to obtain a third inference result. The anomaly judgment unit further performs a third judgment based on the reference information of the evaluation data and the third inference result. The third judgment is a judgment on the error or omission of the third inference result. The notification control unit outputs information related to the training limit of the first model when it is identified that the error or omission of the third inference result is the same as the error or omission of the first inference result.

[0031] Therefore, the assessment of the limits of performance improvement of the inference model through machine learning training does not depend on the skill level of the person responsible for creating the inference model.

[0032] In addition, these general or specific solutions, besides the methods and systems described, can also be implemented by devices, integrated circuits, or computer-readable recording media such as CD-ROMs, or by any combination of devices, systems, integrated circuits, methods, computer programs, and recording media.

[0033] The following description of an information processing method and system based on one aspect of this disclosure is provided with reference to the accompanying drawings. The embodiments shown herein illustrate a specific example of this disclosure. The numerical values, shapes, constituent elements, configurations of constituent elements, connection forms, steps (processes), and sequences of steps shown in the following embodiments are merely examples and are not intended to limit the scope of this disclosure. Furthermore, among the constituent elements in the following embodiments, those not described in the highest-level conceptual solution are shown and are described as arbitrary constituent elements. Additionally, the figures are schematic diagrams and not rigorous illustrations.

[0034] (Implementation Method)

[0035] The information processing method involved in the implementation is a computer-executed method for determining a practical limit (hereinafter also referred to as the training limit). The practical limit refers to an anomaly (omission or error) in the processing results obtained from the inference model trained through machine learning, where the likelihood of performance improvement is low even with further training. The following example illustrating this information processing method is an example of determining the training limit of an inference model for object detection on an image.

[0036] [1. Composition]

[0037] Figure 1 This is a block diagram illustrating a functional configuration example of an information processing system that performs the information processing method according to this embodiment.

[0038] In the information processing system 100 of this embodiment, machine learning training is performed to generate or update an inference model, and the generated or updated inference model is evaluated. Furthermore, based on the evaluation results, the information processing system 100 determines whether the performance improvement through training has reached its limit.

[0039] The information processing system 100 includes one or more computers, each equipped with a processor and a memory, and a server that executes the information processing method. Figure 1 In the example shown, the server, as a functional component, includes: a detection processing unit 10, an anomaly judgment unit 20, an error detection and identification unit 30, an undetected identification unit 40, a notification control unit 50, and a model training unit 60.

[0040] The detection processing unit 10 enables a trained object detection model (hereinafter referred to as the trained model) that has undergone machine learning training to perform object detection processing for performance evaluation. Having obtained evaluation data (image data in this example) for performance evaluation of the trained model, the detection processing unit 10 preprocesses the images used in this performance evaluation. This preprocessing refers to processing performed to prepare for object detection, such as adjusting the image resolution or normalizing pixel values. When obtaining the trained model of the evaluation object, the detection processing unit 10 inputs each preprocessed image into the trained model and outputs the result of the object detection processing (hereinafter referred to as the object detection result), which is the inference processing performed by the object detection model. Specifically, the object detection result is the detection boxes, such as pedestrians and obstacles, detected by the object detection model in the image. Information about the obtained detection boxes is output from the detection processing unit 10 to the anomaly judgment unit 20. This information includes, for example, information about the image to which object detection processing was performed, the extent (position and size) of the detection boxes in the image, and the accuracy (also called likelihood). The detection processing unit 10 is an example of the inference processing unit of this embodiment. The object detection results obtained by the detection processing unit 10 through object detection processing are examples of the first inference result, the second inference result, and the third inference result of this embodiment.

[0041] The anomaly detection unit 20 receives, in addition to the detection box information received from the detection processing unit 10, information about a detection box (hereinafter referred to as the correct answer box) that is shown as a reference for the correct answer to the object detection in the image. Furthermore, the anomaly detection unit 20 determines anomalies in the object detection results performed by the trained model based on the detection box received from the detection processing unit 10 and the detection box showing the correct answer. Anomalies in this example include incorrect detection (error) and undetected (omission). Figure 2 This is a flowchart illustrating an example of the steps of the method for judging these anomalies based on the object detection results performed by the anomaly judgment unit 20.

[0042] The anomaly detection unit 20 performs anomaly detection based on the descending order of accuracy of detection boxes obtained from an image displayed by the information received from the detection processing unit 10. First, the anomaly detection unit 20 determines the detection box with the highest accuracy among the detection boxes for which no anomaly detection has been performed (step S01).

[0043] Next, the anomaly detection unit 20 calculates the IoU (Intersection over Union) between the detection box determined in step S01 and the correct answer box of the image (step S02).

[0044] Next, the anomaly detection unit 20 performs a judgment, determining whether the IoU calculated in step S02 of each detection box with the correct answer box exceeds 0.5 (step S03). If the IoU between the detection box and the correct answer box exceeds 0.5 ("Yes" in step S03), the anomaly detection unit 20 determines that the detection box is a correct detection and pairs it with the correct answer box with an IoU exceeding 0.5 (step S04). If the IoU between the detection box and the correct answer box does not exceed 0.5 ("No" in step S03), the anomaly detection unit 20 determines that the detection box is an incorrect detection (step S05).

[0045] Next, the anomaly detection unit 20 checks whether there are any detection boxes in the detection boxes obtained for this image that have not yet been executed up to S05 (step S06).

[0046] If there are still detection boxes that have not performed these steps ("Yes" in step S06), the exception determination unit 20 returns to step S01.

[0047] If all the detection boxes obtained for the image have been executed up to step S05 (No in step S06), the anomaly determination unit 20 checks whether there is a correct answer box in the correct answer box for the image, and whether the pairing performed in step S04 has been established with any detection box in the steps up to this point (step S07).

[0048] If there is a correct answer box that has not been matched ("Yes" in step S07), the exception judgment unit 20 determines that the correct answer box is due to not being detected (step S08) and ends the series of steps.

[0049] If there are no correct answer boxes that are not matched, in other words, if all correct answer boxes are matched with a certain detection box ("No" in step S07), the series of steps for judging the anomalies in the object detection processing of the image ends.

[0050] The result of the judgment by the anomaly judgment unit 20, in other words, information related to the correctness of each detection box (hereinafter also referred to as error detection information), is output to the error detection recognition unit 30, and information related to non-detection (hereinafter also referred to as non-detection information) is output to the non-detection recognition unit 40. Error detection information and non-detection information will be described later.

[0051] Furthermore, regarding the aforementioned steps, for the sake of simplicity, it is assumed that only the case where a detection box is obtained for a single object detection target will be described. If multiple objects are being detected, step S02 can be performed between the detection box for the same object and the correct answer box. In this case, for example, information related to the type of object being detected can be included in the information of the detection box obtained by the anomaly detection unit 20 and the information of the correct answer box.

[0052] Furthermore, the baseline value of 0.5 for IoU in step S03 is an example, and will be used as an example in the following explanation. This baseline value is set such that, for example, only one of the IoUs calculated between a detection box and each of multiple correct answer boxes exceeds this baseline value. However, even if multiple IoUs exceed the baseline value, the combination of, for example, the detection box with the highest IoU and the correct answer box can be paired in step S04. Alternatively, based on the possibility that the detection result is ambiguous, it can be determined that the detection box is a false detection.

[0053] The detection frame shown by the information received by the anomaly judgment unit 20 from the detection processing unit 10 is an example of the first detection frame in this embodiment. Furthermore, the information of the correct answer frame obtained by the anomaly judgment unit 20 is an example of the reference information in this embodiment, and the correct answer frame is an example of the second detection frame. Furthermore, the anomaly judgment unit 20, according to... Figure 2 The order of judgment shown is an example of the first, second, and third judgments in this embodiment.

[0054] The error detection identification unit 30 obtains error detection information from the anomaly judgment unit 20 and accumulates this error detection information. The error detection information includes, for example, detailed information showing the error detection, such as the image from which object detection was performed, the correctness of the detection result, and the detection bounding box (hereinafter also referred to as the error detection box) of the object detection result judged as an error detection. The error detection identification unit 30 uses the accumulated error detection information to determine whether a newly obtained error detection information detailing an error detection is a recurrence of an error detection that has already occurred. Specifically, the error detection identification unit 30 calculates the Interchange of Unit (IoU) between the newly obtained error detection information detailing the error detection box and the error detection box corresponding to an error detection that occurred when the same image was processed by an older object detection model that has undergone performance evaluation. Furthermore, if the IoU exceeds a predetermined benchmark value (e.g., 0.5), the error detection identification unit 30 identifies the newly obtained error detection information detailing the error detection as the same error detection as that occurring in the older object detection model, and judges it as a recurrence of an error detection that has already occurred. Furthermore, if the IoU is below a predetermined baseline value, the error detection identification unit 30 determines that the error detection details shown in the newly acquired error detection information are not previously observed error detections (new error detections). Information related to this determination is output from the error detection identification unit 30 to the notification control unit 50. This information, in addition to the determination result, includes, for example, information about the image where the error detection occurred. This information is then transmitted to the model training unit 60 (described later) to obtain training data for use in the next machine learning training iteration.

[0055] The undetected detection unit 40 obtains information related to undetected events from the anomaly judgment unit 20 and accumulates this undetected information. The undetected information includes, for example, detailed information showing the undetected event, such as the image from which object detection was performed and the correct answer box (hereinafter referred to as the undetected box) that was judged as undetected. The undetected detection unit 40 uses the accumulated undetected information to determine whether the undetected event showing detailed information newly obtained from the anomaly judgment unit 20 is a recurrence of an already occurring undetected event. Specifically, the undetected detection unit 40 identifies whether the undetected box shown in the newly obtained undetected information is the same as the undetected box corresponding to an undetected event that occurred when the same image was processed by an older object detection model that has undergone performance evaluation. Furthermore, if the two undetected boxes are the same, the undetected detection unit 40 determines that the undetected event showing detailed information in the newly obtained undetected information is a recurrence of an already occurring undetected event. Conversely, if the two undetected boxes are different, the undetected detection unit 40 determines that the undetected event showing detailed information in the newly obtained undetected information is not a previously occurring undetected event (a new undetected event). Furthermore, information related to the result of this judgment is output from the undetected recognition unit 40 to the notification control unit 50. This information, in addition to the judgment result itself, includes, for example, information about the occurrence of new undetected images. This information is then transmitted to the model training unit 60 (described later) to obtain training data for the next execution of machine learning training.

[0056] The notification control unit 50 determines whether a discrepancy has occurred based on the received information related to the judgment result, and sends a notification corresponding to the judgment result to the model training unit 60. The information related to the judgment result is input from the error detection and recognition unit 30 and the non-detection recognition unit 40. Here, a discrepancy refers to an error detection or non-detection that has been eliminated through machine learning training, recurring after further training to eliminate error detection or non-detection in the trained model. As a specific example of notification, if a discrepancy is determined to have occurred, the notification control unit 50 notifies the user of the information processing system 100, in other words, the creator of the object detection model, that the training limit has been reached, as information related to the training limit. As information related to the training limit, the detection result (detection box) related to the discrepancy can be output. This notification is, for example, delivered via an image display on a display device (not shown), which is configured with... Figure 1 The computer connection to the server. Furthermore, if it is determined that no reversal has occurred, the notification control unit 50 can notify the user that the training has normally ended. These notifications can be made, for example, via sound output from a speaker (not shown), which is connected to the server's computer. Figure 1The computer connection to the server. Furthermore, if it is determined that no reversal has occurred, the control unit 50 is notified to output the information related to new error detection or new undetected information received from the error detection and identification unit 30 and the undetected information identification unit 40 to the model training unit 60.

[0057] The model training unit 60 performs machine learning training to obtain an object detection model that serves as the subject of performance evaluation. When receiving input from the notification control unit 50 regarding information related to a new false detection, the model training unit 60 obtains training data from training data D20 for training to eliminate that false detection. The training data used for training to eliminate false detection is, for example, data of an image where the new false detection occurred, as determined by the information related to the new false detection. When receiving input from the notification control unit 50 regarding information related to a new undetected image, the model training unit 60 obtains training data from training data D20 for training to eliminate that undetected image. The training data used for training to eliminate undetected images is, for example, data of an image where the new undetected image occurred, as determined by the information related to the new undetected image. In addition to training data D20, data based on evaluation data D10 is used for machine learning training. Furthermore, as training data used for training to eliminate false detections, data of images similar to the image where the new false detection occurred can be used. Similarly, as training data used for training to eliminate undetected images, data of images similar to the image where the new undetected image occurred can be used. Similar images are those with the same or similar composition, but taken with different photographic settings such as exposure or shutter speed. Other examples of similar images include those that have undergone image processing to add or reduce noise, as well as images whose brightness, contrast, chroma, or white balance has been altered.

[0058] To perform machine learning training, the model training unit 60 prepares a data set that combines the acquired training data with training data used in training the already trained model to detect objects with incorrect or no detections. The model training unit 60 then uses this data set to train the object detection model with the specified configuration. At the end of training, a new trained model is output from the model training unit 60.

[0059] These functional components are implemented by the computer that constitutes the server, through the processor executing programs stored in memory, or by reading data stored in memory as needed.

[0060] In addition, Figure 1For example, in information processing system 100, evaluation data D10 and training data D20 are input to the server from outside the server, but one or both can be set within the server.

[0061] [2. Action]

[0062] Next, the information processing method executed in the information processing system 100 described above will be explained. Through this information processing method, the performance of the inference model generated or updated through machine learning training is evaluated, and based on the evaluation result, it is determined whether the limit for improving performance through training has been reached. Figure 3 This is a flowchart illustrating an example of the steps of this information processing method. Furthermore, to facilitate understanding of the outline of this information processing method, a portion of the steps may be simplified, or the entire method may be described along the assumed path, with supplementary explanations provided later. Figure 3 The example shown is that, in the process of improving the accuracy of an object detection model, image data for evaluation is input to the trained model to obtain object detection results. Based on the object detection results and reference information from the image data for evaluation, the anomaly detection unit 20 performs anomaly detection (reference). Figure 2 The steps that begin when you obtain the result. Here, Figure 3 Before the steps shown, the object detection model that has been trained to perform object detection by inputting evaluation data is an example of the first model in this embodiment. The object detection result output by the object detection model is an example of the first inference result. The anomaly judgment performed by the anomaly judgment unit 20 on the object detection result is an example of the first judgment.

[0063] In the result of this first determination, the error detection identification unit 30, which receives error detection information from the anomaly determination unit 20, compares this error detection information with previously received accumulated error detection information to determine whether a new error detection has occurred during the object detection processing performed for the evaluation of the first model (step S10). During the period when no new error detection occurs ("No" in step S10), the information processing method does not proceed to the next step. In other words, during the period when the same error detection is identified again, for example, an operation to eliminate the error detection is performed.

[0064] In the event of a new false detection ("Yes" in step S10), the information processing method proceeds to step S11. In step S11, the model training unit 60, which receives information related to the result of the judgment from the false detection identification unit 30 via the notification control unit 50, obtains a new object detection model through machine learning training. In this training, the following data is used as training data: existing training data used in training the object detection model where the new false detection (hereinafter referred to as false detection A) occurred, and data of the image of the evaluation data where false detection A occurred, and data of images similar to that image, or both. The object detection model obtained in step S11 is hereinafter referred to as model a. The training performed by the model training unit 60 in step S11 is an example of the first training in this embodiment, and model a is an example of the second model in this embodiment. Furthermore, the image data input to model a as evaluation data, including the image of the false detection, in other words, the image data of the image where false detection A occurred, is an example of the first evaluation data in this embodiment.

[0065] Next, the detection processing unit 10 inputs image data as evaluation data into model a, and obtains the object detection result output from model a for the input (step S20). The object detection result output by model a is an example of the second inference result in this embodiment.

[0066] Next, the anomaly detection unit 20, based on the reference information of the evaluation data input to model a in step S20 and the object detection result output by model a, performs the following steps: Figure 2 The abnormality determination shown in step S21 is an example of the second determination in this embodiment.

[0067] Next, in the result of the second determination, the error detection identification unit 30, which receives error detection information from the anomaly determination unit 20, checks the error detection information against previously received accumulated error detection information to identify whether it is the same as error detection A, in other words, to confirm whether error detection A has been eliminated (step S22). If error detection A has not been eliminated ("No" in step S22), the process restarts from step S11. In this case, step S11 is executed by changing, for example, the training data used in the previous execution of step S11, or hyperparameters in machine learning training such as batch size, iteration number, or epoch number.

[0068] If false detection A is eliminated (Yes in step S22), in the result of the second determination, the undetected information identification unit 40, which receives undetected information from the anomaly determination unit 20, checks the undetected information against the accumulated undetected information received in the past, and determines whether a new undetected event has occurred in the object detection processing performed for evaluating model a (step S23). If a new undetected event has occurred (Yes in step S23), information related to the result of this determination is transmitted from the undetected information identification unit 40 to the model training unit 60 via the notification control unit 50. The model training unit 60 further obtains a new object detection model based on machine learning training (step S24). In this training, the following data is used as training data: the existing training data used in the training of model a in step S11 and the evaluation data of the image where the new undetected event occurred (hereinafter referred to as undetected event B), i.e., the image data and the data of images similar to the image. The object detection model obtained in step S24 is hereinafter simply referred to as model b. The training performed in the model training unit 60 in step S24 is an example of the second training in this embodiment, and model b is an example of the third model in this embodiment. Furthermore, the image data input to model b as evaluation data includes data that detects missed images, or in other words, data that shows images of model b not being detected; this is an example of the second evaluation data in this embodiment.

[0069] Next, the detection processing unit 10 inputs image data as evaluation data into model b and obtains the object detection result output from model b in response to the input (step S30). The object detection result output by model b is an example of the third inference result in this embodiment.

[0070] Next, the anomaly detection unit 20, based on the reference information of the evaluation data input to model b in step S30 and the object detection result output by model b, performs the following steps: Figure 2 The abnormality determination in step S31 is an example of the third determination in this embodiment.

[0071] Next, in the result of the third determination, the undetected information identification unit 40, which receives undetected information from the anomaly determination unit 20, checks the undetected information against the previously received accumulated undetected information to confirm whether undetected information B has been eliminated (step S32). If undetected information B has not been eliminated ("No" in step S32), step S24 is repeated. In this case, step S24 is executed by changing, for example, the training data used in the previous execution of step S24, or hyperparameters in machine learning training such as batch size, iteration number, or epoch number.

[0072] If error B is not detected ("Yes" in step S32), in the result of the third determination, the error detection identification unit 30, which receives error detection information from the anomaly determination unit 20, determines whether an error detection has occurred based on the error detection information (step S33). If an error detection has occurred ("Yes" in step S33), it is checked against previously received accumulated error detection information to further determine whether the error detection that occurred in model b is a recurrence of the eliminated error detection A (step S40). The result of the determination in step S40 is input to the notification control unit 50. If error detection A recurs ("Yes" in step S40), the notification control unit 50 outputs information related to the training limit, for example, outputting specified data showing this information to a display device or speaker to notify the operator who created the object detection model (step S50). The information related to the training limit, for example, indicates the probability of reaching the training limit. The operator who receives this notification can know that the performance of the model being trained through machine learning has been improved and the limit has been reached.

[0073] In addition, Figure 3 In the example, false detection A is eliminated in the subsequently trained model a ("Yes" in step S22), and if no undetection occurs in model a ("No" in step S23), the user is notified that no backlash has occurred (step S51). Similarly, undetection B that occurs in model a is eliminated in the subsequently trained model b ("Yes" in step S32), and if no false detection occurs in model b ("No" in step S33), the user is notified that no backlash has occurred, or that the training limit has not been reached (step S51).

[0074] Furthermore, the described sequence of steps is an example, and the processing content of the information processing method involved in this embodiment is not limited to this example. For example, in the example described, the condition for determining that the training limit has been reached is the occurrence of the following phenomenon: when eliminating an error detection A that has already occurred, a non-detection B occurs, and when eliminating non-detection B, the eliminated error detection A occurs again. The content of such a phenomenon used as the condition for determining that the training limit has been reached is not limited to this, as long as such a contradiction occurs where multiple anomalies cannot be eliminated simultaneously. For example, if the error detection and non-detection are replaced in the example described, and an error detection occurs when eliminating a non-detection that has already occurred, and when eliminating the error detection, the eliminated non-detection occurs again, it can be determined that the training limit has been reached. In addition, the content of the contradiction in determining that the training limit has been reached can be the occurrence of the following phenomenon: the existence of multiple error detections that cannot be eliminated simultaneously, or the existence of multiple non-detections that cannot be eliminated simultaneously.

[0075] (Variations and other supplementary matters)

[0076] The information processing methods involved in one or more solutions of this disclosure are not limited to the description of the embodiments. Examples of various modifications that can be conceived by those skilled in the art to the embodiments are also included in the solutions of this disclosure without departing from the spirit of this disclosure. Such modifications, as well as other supplementary matters to the description of the embodiments, are described below.

[0077] (1) In the description of the embodiment, IoU was used as the evaluation metric for whether the object detection result output by the object detection model is a false detection, but the evaluation method for the object detection result is not limited to this. In addition to IoU, the following are examples of evaluation metrics that can be used by the information processing method of this disclosure as obtained based on the detection box and the correct answer box: Precision (also known as accuracy, used for false detection), Recall (also known as recall, used for false omission), and F-score (used for the evaluation of false and false detections).

[0078] (2) In the description of the above embodiment, the notification control unit 50 outputs data displayed on a display device or reproduced on a speaker in order to make a notification. This is an example of the form of output from the information processing system 100, but other forms are also possible. For example, data may be output from a storage device (not shown) provided by the information processing system 100 or external to the information processing system 100, which is data stored as a log related to the training of machine learning in the information processing system 100. Furthermore, as another example, data that is the object of processing or reference data for some kind of information processing inside or outside the information processing system 100 may be output. For example, it may be used in information processing that determines hyperparameters, which are parameters used to create an inference model.

[0079] (3) In the described embodiment, the machine learning training for obtaining model a in S11 and model b in step S24 can be performed using full training data to create each model from scratch. In this case, the first model, the second model, and the third model are obtained as independent models. Alternatively, additional training data can be used to supplement the existing models. Regarding the latter... Figure 3 The specific example is as follows: In order to obtain model b in step S24, additional training is performed on model a using one or both of the data of undetected images and data of images similar to those images from object detection by model a. Furthermore, in this case, the updated first model is obtained as the second model, and the updated second model is obtained as the third model.

[0080] (4) In the described embodiment, an example of a reversal is given as follows: regarding a certain anomaly (undetected or falsely detected), after it is eliminated, another anomaly (undetected or falsely detected) occurs in between, and then occurs again. However, the anomaly occurring between these recurring anomalies is not limited to one. For example, when repeating a loop that includes training of machine learning performed to obtain a desired model, object detection, and anomaly judgment of object detection results, information related to the content of anomalies occurring in each loop is accumulated, and in the case of anomalies occurring in each loop, the content is checked against each anomaly. The result of this check is that the content of the anomaly occurring in the latest loop is the same as the anomaly occurring in any previous loop, which also includes a reversal in this disclosure. In this sense, the first inference result, the second inference result, and the third inference result of this disclosure are not limited to being obtained consecutively, and any one of them can be obtained in that order on the time axis during repeated loops.

[0081] (5) In the described embodiment, an example of an inference model for object detection using image data is used for illustration. However, the object processed by the inference model is not limited to image data. Specific examples of sensor data other than image data include sound, distance points, pressure, temperature, humidity, odor, etc., or data other than sensor data, such as speech data. Moreover, the inference model can output results such as voice recognition, speaker recognition, and stylistic analysis when this data is input.

[0082] (6) Some or all of the functional components of the aforementioned information processing systems can be constituted by a single system LSI (Large Scale Integration). A system LSI is a multifunctional LSI manufactured by integrating multiple components on a single chip. Specifically, it is a computer system consisting of a microprocessor, ROM (Read-Only Memory), RAM (Random Access Memory), etc. The computer program is stored in the ROM. The microprocessor operates according to the computer program, thereby enabling the system LSI to achieve the functions of each component.

[0083] Furthermore, a system LSI is used here, but depending on the level of integration, it can also be called an IC, LSI, very large LSI, or extremely large LSI. In addition, the method of integrated circuitization is not limited to LSI; it can be implemented using dedicated circuits or general-purpose processors. Programmable FPGAs (Field Programmable Gate Arrays) or reconfigurable processors that connect and configure the internal circuitry of the LSI can be used after the LSI is fabricated.

[0084] Furthermore, with advancements in semiconductor technology or the emergence of other derived technologies, when integrated circuit technologies emerge that can replace LSIs, these technologies can certainly be used for the integration of functional blocks. This may also be applicable to technologies such as biotechnology.

[0085] (7) One aspect of this disclosure is not limited to the use of Figure 2 or Figure 3 The information processing method described above in the flowchart can be a program executed by a computer, or an information processing system including a computer. Furthermore, one embodiment of this disclosure can be a computer-readable, non-temporary recording medium that records a computer program.

[0086] The information processing methods disclosed herein can be used to develop models that improve performance through machine learning training.

[0087] Symbol Explanation

[0088] 10. Testing and Processing Department

[0089] 20. Anomaly Detection Department

[0090] 30 Error Detection and Recognition Department

[0091] 40 Undetected Identification Unit

[0092] 50 Notification to the Control Department

[0093] 60 Model Training Department

[0094] 100 Information Processing System

[0095] D10 Evaluation Data

[0096] D20 training data

Claims

1.An information processing method executed by a computer, in the information processing method, evaluation data is obtained, the evaluation data being image data; a first model that performs an inference process is input with the evaluation data, and a first inference result is obtained; based on reference information of the evaluation data and the first inference result, a first determination is performed, the first determination being a determination of an error or an omission of the first inference result; a second model is obtained through first training, the first training being training of machine learning in which one or both of first evaluation data and data similar to the first evaluation data are included in training data, the first evaluation data being the evaluation data corresponding to the first inference result that is determined to have an error or an omission; the second model is input with the evaluation data, and a second inference result is obtained; based on reference information of the evaluation data and the second inference result, a second determination is performed, the second determination being a determination of an error or an omission of the second inference result; a third model is obtained through second training, the second training being training of machine learning in which second evaluation data and data similar to the second evaluation data are included in training data, the second evaluation data being the evaluation data corresponding to the second inference result, the second inference result having an error or an omission that is not present in the first inference result; the third model is input with the evaluation data, and a third inference result is obtained; based on reference information of the evaluation data and the third inference result, a third determination is performed, the third determination being a determination of an error or an omission of the third inference result; and in a case where the error or the omission present in the third inference result is identified as being the same as the error or the omission present in the first inference result, information about a training limit of the first model is output. 2.The information processing method according to claim 1, the inference process is an object detection process, the first inference result, the second inference result, and the third inference result are each an object detection result. 3.The information processing method according to claim 2, the object detection result includes a first detection frame, the first detection frame being a detection frame of an object as a result of the object detection process, the reference information includes a second detection frame, the second detection frame being a detection frame of an object as a reference, the error or the omission is determined based on the first detection frame and the second detection frame. 4.The information processing method according to any one of claims 1 to 3, a notification is performed by outputting the information, the notification indicating a possibility that the training limit has been reached, the notification is performed via an image or a sound. 5.An information processing system comprising: an inference processing unit; an abnormality determination unit; a model training unit; and a notification control unit, ​ The inference processing section obtains evaluation data, inputs the evaluation data to a first model that performs inference processing, obtains a first inference result, and the evaluation data is image data, The abnormality determination section performs a first determination based on the reference information of the evaluation data and the first inference result, and the first determination is a determination of an error or omission of the first inference result, The model training section obtains a second model by first training, and the first training is a training of machine learning in which one or both of first evaluation data and data similar to the first evaluation data are included in training data, and the first evaluation data is the evaluation data corresponding to the first inference result determined to have an error or omission, The inference processing section further inputs the evaluation data to the second model and obtains a second inference result, The abnormality determination section further performs a second determination based on the reference information of the evaluation data and the second inference result, and the second determination is a determination of an error or omission of the second inference result, The model training section further obtains a third model by second training, and the second training is a training of machine learning in which one or both of second evaluation data included in the evaluation data and data similar to the second evaluation data are included in training data, and the second evaluation data is the evaluation data corresponding to the second inference result, and the second inference result has an error or omission that the first inference result does not have, The inference processing section further inputs the evaluation data to the third model and obtains a third inference result, The abnormality determination section further performs a third determination based on the reference information of the evaluation data and the third inference result, and the third determination is a determination of an error or omission of the third inference result, The notification control section outputs information about a training limit of the first model in a case where an error or omission that the third inference result has is identified as being the same as an error or omission that the first inference result has.

Images