Ai video analysis system, and method and program for improving precision of ai thereof

By comparing AI model results with human analysis and feeding back undetected and false positive events, the AI video analysis system improves accuracy by reducing false positives and undetected events.

JP2024166963AInactive Publication Date: 2024-11-29TOYOTA JIDOSHA KK
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023083412
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing AI video analysis systems suffer from inaccuracies in event detection, leading to false positives and undetected events, necessitating improved accuracy.

Method used

A data pipeline is constructed to compare AI model analysis results with human analysis results, providing feedback to the AI model with undetected and false positive events for enhanced learning.

Benefits of technology

This approach enhances AI accuracy by reducing false positives and undetected events through efficient learning from human feedback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024166963000001_ABST
    Figure 2024166963000001_ABST
Patent Text Reader

Abstract

To improve precision of AI of an AI video analysis system.SOLUTION: At least one processor which constitutes an Ai video analysis system 10 executes the following processes. A first process is to acquire an AI model 12 so trained as to analyze videos. A second process is to acquire camera video data captured by a camera 11. A third process is to cause the AI model 12 to analyze the camera video data. A fourth process is to acquire an analysis result of the camera video data by a person 13. Then a fifth process is to impart to the Ai model 12 feedback based upon a comparison between the analysis result of the camera video data by the AI model 12 and the analysis result of the camera video data by the person 13.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an artificial intelligence (AI) video analysis system, a method for improving the accuracy of AI in an AI video analysis system, and a program for an AI video analysis system. [Background technology]

[0002] International Publication No. 2019 / 151411 discloses that when generating learning pattern data corresponding to the result of deep learning processing using image data, external data corresponding to the image data is acquired from outside. When the external data includes visual recognition data corresponding to a visual recognition behavior of a person included in the activity of the person involved in generating output information using the learning pattern data, learning pattern data corresponding to the person's expertise can be generated.

[0003] Incidentally, today, the use of AI in video analysis systems is being considered. In conventional video analysis systems under consideration, video data for learning is collected in advance, and the data is checked and tagged by humans to allow the AI ​​to learn from it. However, this method poses a problem in terms of the accuracy of the AI ​​that is trained. This is a common problem with the method described in the above publication, and there is a demand for further improvements in the accuracy of the AI.

[0004] In addition to WO 2019 / 151411, JP 2010-518416 A and WO 2022 / 059123 A can be exemplified as documents showing the technical state of the technical field related to the present disclosure. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2019 / 151411 [Patent Document 2] JP 2010-518416 A [Patent Document 3] International Publication No. 2022 / 059123 Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure has been made in view of the above problems. One objective of the present disclosure is to improve the accuracy of AI in an AI video analysis system. [Means for solving the problem]

[0007] The present disclosure provides an AI video analysis system for achieving the above object. The AI ​​video analysis system of the present disclosure includes a memory that stores a plurality of instructions, and at least one processor communicatively coupled to the memory. The plurality of instructions are configured to cause the at least one processor to execute at least the following five processes. The first process is to obtain an AI model trained to analyze video data. The second process is to obtain camera video data captured by a camera. The third process is to cause the AI ​​model to analyze the camera video data. The fourth process is to obtain a result of a human analysis of the camera video data. And the fifth process is to provide feedback to the AI ​​model based on a comparison between the result of the analysis of the camera video data by the AI ​​model and the result of the analysis of the camera video data by the human. Note that providing feedback to the AI ​​model may include providing a set of an undetected event that was detected by a human analysis but not detected by the AI ​​model analysis, and camera video data in which a human detected the undetected event. In addition, providing feedback to the AI ​​model may include providing the AI ​​model with a set of false positive events that were detected by analysis by the AI ​​model but not by human analysis, and camera video data in which the AI ​​model detected the false positive events.

[0008] The present disclosure provides a method for achieving the above object. The method of the present disclosure is a method for improving the accuracy of AI in an AI video analysis system, and includes at least the following five steps. The first step is to obtain an AI model trained to analyze video data. The second step is to obtain camera video data captured by a camera. The third step is to have the AI ​​model analyze the camera video data. The fourth step is to obtain the results of a human analysis of the camera video data. And the fifth step is to provide feedback to the AI ​​model based on a comparison between the results of the AI ​​model's analysis of the camera video data and the results of a human analysis of the camera video data.

[0009] The present disclosure provides a program for achieving the above object. The program of the present disclosure is a program including a plurality of instructions executed by at least one processor. The plurality of instructions are configured to cause the at least one processor to execute at least the five processes described above. The program of the present disclosure may be recorded on a computer-readable recording medium. Effect of the Invention

[0010] According to the technology disclosed herein, a data pipeline is constructed in which the results of analysis of camera video data by an AI model and the results of analysis of camera video data by humans are compared and fed back to the AI ​​model, thereby enabling efficient learning of the AI ​​model and contributing to improving the accuracy of the AI. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing a configuration of an AI video analysis system according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a hardware configuration of an AI video analysis system. [Diagram 3] FIG. 1 is a diagram showing an example of processing in an AI video analysis system applied to security. [Figure 4]FIG. 13 is a diagram showing an example of a method for tagging video data. [Diagram 5] FIG. 13 is a diagram showing an example of a flow of determining feedback to an AI model. [Figure 6] FIG. 1 is a diagram showing an example of processing in an AI video analysis system applied to early childhood education. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] 1. AI video analysis system FIG. 1 shows the configuration of an AI video analysis system 10 according to this embodiment. The AI ​​video analysis system 10 is a system that analyzes video data using AI and detects noteworthy events contained in the video data. The AI ​​video analysis system 10 is also a system in which a method for improving the accuracy of AI according to this embodiment is implemented. The video data to be analyzed in the AI ​​video analysis system 10 is video data (also referred to as camera video data) captured by a camera 11. The camera 11 is, for example, a network camera installed in a facility to be monitored.

[0013] The AI ​​video analysis system 10 analyzes video data using an AI model 12. The AI ​​model 12 is a machine learning model trained using tagged video data as teacher data. Tagging is performed on video data that contains noteworthy events. The AI ​​video analysis system 10 inputs video data captured by a camera 11 into the AI ​​model 12, and detects noteworthy events contained in the video data through analysis by the AI ​​model 12.

[0014] The analysis results by the AI ​​model 12, i.e., the events detected by the AI ​​model 12, are compared and judged by the comparison and judgment unit 14. The analysis results by the AI ​​model 12 are compared with the analysis results by the person 13. The person 13 in the AI ​​video analysis system 10 is, for example, a person responsible for analyzing camera images. In the AI ​​video analysis system 10, the person 13 analyzes the same video data as the video data input to the AI ​​model 12. The analysis results by the person 13 are input into the comparison and judgment unit 14 together with the analysis results by the AI ​​model 12.

[0015] The comparison and judgment unit 14 regards the analysis result by the person 13 as the correct answer and judges whether the analysis result by the AI ​​model 12 is correct. If an event is detected by the AI ​​model 12 and also by the person 13, the analysis result by the AI ​​model 12 is judged to be correct. If an event is not detected by the AI ​​model 12 and also by the person 13, there is no input to the comparison and judgment unit 14. If there is no input from either the AI ​​model 12 or the person 13, the analysis results of both match, so it is judged that the AI ​​model 12 has obtained a correct analysis result.

[0016] If the analysis results by the AI ​​model 12 and the analysis results by the human 13 do not match, the comparison and judgment unit 14 feeds back the result of the comparison and judgment to the AI ​​model 12. There are two cases in which the two analysis results do not match. The first case is when an event is detected by the AI ​​model 12, but the event is not detected by the human 13. In this case, the detection of the event by the AI ​​model 12 is a false positive, so the analysis result by the AI ​​model 12 is judged to be incorrect. The comparison and judgment unit 14 feeds back to the AI ​​model 12 a set of the event that was falsely detected by the AI ​​model 12 (false positive event) and the video data in which the AI ​​model detected the false positive event, as learning data.

[0017] The second case in which the two judgments do not match is when an event not detected by AI model 12 is detected by human 13. In this case, since the event that should be detected by AI model 12 is not detected, the analysis result by AI model 12 is judged to be erroneous. Comparison and judgment unit 14 feeds back to AI model 12 a set of the event not detected by AI model 12 (undetected event) and video data in which human 13 detects the undetected event as learning data.

[0018] The comparison and judgment unit 14 feeds back the learning data to the AI ​​model 12 every time learning data is generated, every fixed period, or every time a fixed amount of learning data is accumulated. Since such a data pipeline is built in the AI ​​video analysis system 10, efficient learning of the AI ​​model 12 is possible.

[0019] Incidentally, human-in-the-loop machine learning (HIML) is known as a form of machine learning. The feedback of learning data to the AI ​​model 12 by the AI ​​video analysis system 10 is also a type of HIML. However, in existing HIML, a human judges errors in the AI ​​output, labels the erroneous data, and makes the AI ​​relearn it. In other words, existing HIML teaches the AI ​​false positive data that the AI ​​has erroneously detected. In contrast, the AI ​​video analysis system 10 can feed back not only false positive data but also undetected data to the AI ​​model 12 by comparing the analysis results by the AI ​​model 12 and the analysis results by a human 13. As a result, it is possible to further improve the accuracy of the AI ​​model 12 compared to existing HIML, and it is possible to reduce not only false positives but also undetected events of noteworthy events.

[0020] The AI ​​model 12 and the comparison and judgment unit 14 can be provided in the server 20. In this case, the video data captured by the camera 11 is input to the server 20 via a network and processed by the server 20. Alternatively, although not shown in the figure, the camera 11 may be configured as an AI network camera equipped with the AI ​​model 12 and the comparison and judgment unit 14, and the video data may be processed in the camera 11.

[0021] Fig. 2 shows an example of the hardware configuration of the server 20 when the AI ​​video analysis system 10 includes the server 20. In the example shown in Fig. 2, the server 20 includes a processor 21, a memory 22, a drive 25, a communication module 27, and a user interface 28. These elements are connected by wire to enable communication. Note that the number of processors 21 is not limited to one.

[0022] The memory 22 stores a program 23 and parameters 24. The program 23 is a program for constructing the above-mentioned data pipeline. The program 23 is composed of a plurality of instructions executable by the processor 21. The instructions include instructions for the processor 21 to acquire a function as the AI ​​model 12 and instructions for the processor 21 to acquire a function as the comparison and judgment unit 14. The parameters 24 are parameters used when the program 23 is executed, and include parameters learned by the AI ​​model 12.

[0023] The drive 25 includes a storage medium 26. The storage medium 26 is a non-transitory computer-readable storage medium that stores learning data for training the AI ​​model 12. The storage medium 26 is also used to store the program 23. The program 23 may be provided via a network. The communication module 27 is provided for communication with external devices including the camera 11. The external devices also include terminal devices for displaying video data to the person 13. The communication module 27 may be a wireless communication module or a wired communication module. The user interface 28 is provided for acquiring data input by the person 13.

[0024] 2. Specific application example of AI video analysis system 1 The above-mentioned AI video analysis system 10 can be applied to security guards by security companies. Figure 3 is a diagram showing an example of processing in the AI ​​video analysis system 100 applied to security guards. The AI ​​video analysis system 100 is composed of a security AI 100, a security system 120, a monitor 130, and a security guard 140. The processing performed by the AI ​​video analysis system 100 will be described below.

[0025] The security AI 100 performs a monitoring process 111 that constantly monitors camera footage, and anomaly detection process 112 that detects anomalies from the video data. Video data is captured by a camera installed at a location to be guarded. In the anomaly detection process 112, an event defined as an anomaly is detected by video analysis using an AI model. The anomaly detected by the security AI 100 is notified to the security system 120. The anomaly notification output to the security system 120 includes the camera image in which the anomaly was detected, and data identifying the event detected as an anomaly in the camera image.

[0026] The security system 120 performs a monitoring process 121 for monitoring camera footage, and anomaly detection process 122 for detecting anomalies from the video data. The monitoring process 121 and the anomaly detection process 122 are performed by a person in charge of the security system 120. If the monitor 130 is a human, the person who performs the monitoring process 121 and the anomaly detection process 122 may be the same person as the monitor 130 or a different person. If an anomaly is detected by the anomaly detection process 122, an anomaly notification is output, which includes a camera image in which the anomaly was detected, and data identifying the event detected as an anomaly in the camera image.

[0027] In the security system 120, an abnormality notification reception process 123 is performed to receive the abnormality notification output by the abnormality detection process 112 and the abnormality notification output by the abnormality detection process 122. The abnormality notification received by the abnormality notification reception process 123 is sent to a monitor 130. The monitor 130 performs a dispatch necessity determination process 131 to determine the need to dispatch a security guard 140. In the dispatch necessity determination process 131, it is determined whether the abnormality notification output from the security AI 100 is a false alarm, and if the abnormality notification is not a false alarm, it is determined that it is necessary to dispatch a security guard 140. The monitor 130 who performs the dispatch necessity determination process 131 may be a human or a programmed system.

[0028] If the abnormality notification output from the security AI 100 is determined to be a false alarm in the dispatch necessity determination process 131, the security system 120 performs false alarm processing 124. In false alarm processing 124, the contents of the abnormality notification determined to be a false alarm, i.e., the camera image in which the abnormality was erroneously detected and data specifying the event in the camera image that was erroneously detected as an abnormality, are output to the security AI 100.

[0029] If it is determined in the dispatch necessity determination process 131 that it is necessary to dispatch the security guard 140, the security system 120 performs a response instruction process 125. In the response instruction process 125, the security system 120 instructs the security guard 140 to dispatch to the scene.

[0030] In response to response instruction process 125 by the security system 120, the security guard 140 performs response / report process 141. In response / report process 141, the security guard 140 responds to the detected abnormality, and the security guard 140 reports the response result to the security system 120. The response result reported to the security system 120 includes, for example, the content of the event detected as an abnormality and the content of the response to the abnormality.

[0031] In response to response / report process 141 by the security guard 140, the security system 120 performs report receipt process 126. In the report receipt process 126, tagging of video data is performed using the report from the security guard 140. In tagging, data that identifies the event detected as an anomaly in the camera image is added to the camera image in which an anomaly was detected.

[0032] FIG. 4 is a diagram showing an example of a method for tagging video data. Tagging of video data can be performed using the video management system 105. In the example shown in FIG. 4, a security guard 140 who is dispatched in response to an instruction from the security system 120 reports the details of the abnormality using the wireless terminal 102. There are three types of abnormalities to be detected: "snatching," "falling," and "intrusion." To report these three abnormalities, the wireless terminal 102 is provided with three buttons. "Snatching" is assigned to button 1, "falling" is assigned to button 2, and "intrusion" is assigned to button 3.

[0033] When the security guard 140 presses any button on the wireless terminal 102, a signal specific to the pressed button is output from the wireless terminal 102. The signal output from the wireless terminal 102 is received by the wireless receiver 103 and input to the video management system 105 via the I / O module 104. At the same time, video data from the camera 101 is also input to the video management system 105. The video management system 105 creates metadata 107 using the data included in the signal input from the wireless terminal 102, and links the metadata 107 to the video data 106 based on the timestamp. In this manner, the video data is tagged. The tagged video data is stored in a storage device provided in the security system 120.

[0034] 3 to continue the explanation of the report reception process 126. In the report reception process 126, the video data including undetected events that could not be detected as anomalies by the AI ​​model among the tagged video data, and data identifying the undetected events are output to the security AI 100.

[0035] The security AI 100 performs a feedback process 113 to the AI ​​model. In the feedback process 113, data input from the security system 120 by the false alarm process 124 and data input from the security system 120 by the report reception process 126 are fed back to the AI ​​model as learning data.

[0036] The determination of feedback to the AI ​​model in the AI ​​video analysis system 100 can be represented, for example, by the flow shown in Figure 5. According to this flow, first, the process of step S1 is performed. In step S1, it is determined whether an abnormality has been detected by the AI ​​model. The process of step S1 corresponds to the abnormality detection process 112.

[0037] If the AI ​​model detects an abnormality, the flow proceeds to step S2, and if the AI ​​model does not detect an abnormality, the flow proceeds to step S3. In steps S2 and S3, it is determined whether an abnormality has been detected by a person in charge of the security system 120. The processes in steps S2 and S3 correspond to the abnormality detection process 122.

[0038] If an abnormality is detected by the AI ​​model and also by a human, the flow proceeds to step S4. If an abnormality is not detected by the AI ​​model and also by a human, the flow proceeds to step S9. The processes of steps S4 and S9 correspond to the dispatch necessity determination process 131. In step S4, it is determined that the dispatch of security guards is necessary. In step S9, it is determined that the dispatch of security guards is not necessary. In these cases, the analysis results by the AI ​​model are correct in light of the analysis results by a human, so feedback to the AI ​​model, i.e., re-learning of the AI ​​model, is not required.

[0039] If an abnormality is detected by the AI ​​model but not by a human, the flow proceeds to step S5. If an abnormality is not detected by the AI ​​model but is detected by a human, the flow proceeds to step S7. The processes of steps S5 and S7 correspond to the dispatch necessity determination process 131. In step S5, it is determined that dispatch of security guards is unnecessary. In step S7, it is determined that dispatch of security guards is necessary. In these cases, the analysis results by the AI ​​model are incorrect compared to the analysis results by a human, so feedback is provided to the AI ​​model, i.e., the AI ​​model is re-learned.

[0040] If an abnormality is detected by the AI ​​model but not by a human, the process of step S6 is performed as feedback to the AI ​​model. If an abnormality is not detected by the AI ​​model but is detected by a human, the process of step S8 is performed as feedback to the AI ​​model. The processes of steps S6 and S8 correspond to feedback process 113. In step S6, a set of a false positive event that has been falsely detected as an abnormality by the AI ​​model and video data in which the AI ​​model has detected the false positive event is fed back to the AI ​​model. In step S8, a set of an undetected event that has not been detected as an abnormality by the AI ​​model and video data in which a human has detected the undetected event is fed back to the AI ​​model.

[0041] 3. Specific application example 2 of the AI ​​video analysis system The AI ​​video analysis system 10 can also be applied to early childhood education in the education industry. Figure 6 is a diagram showing an example of processing in the AI ​​video analysis system 200 applied to early childhood education.

[0042] In the AI ​​video analysis system 200, a monitoring process 211 for monitoring the video of the camera is performed by the server. The camera is installed in the classroom where the child to be monitored is studying. The video data of the camera acquired by the monitoring process 211 is processed by an AI model 212 provided in the server. In the AI ​​model 212, an observation process 213 for observing the concentration phenomenon of the child from the video data and an observation process 214 for observing the child's behavior other than the so-called "work time" from the video data are performed. Furthermore, in the AI ​​model 212, an estimation process 215 for estimating the developmental level of the child based on the observation results obtained in the two types of observation processes 213 and 214 is performed. The timing to intervene in the child's behavior is calculated by the observation processes 213 and 214 and the estimation process 215 by the AI ​​model 212. The intervention timing calculated by the AI ​​model 212 is recorded as evidence by a recording process 216.

[0043] Meanwhile, a monitoring process 221 for monitoring the camera image and a detection process 222 for detecting an event requiring intervention are performed by the teacher educating the child. In the detection process 222, the teacher observes the child's behavior by watching the camera image and detects an event requiring intervention from the child's behavior in the image.

[0044] In the AI ​​video analysis system 200, a judgment process 217 and a feedback process 218 are performed by the server. The judgment process 217 is a process for judging whether the intervention timing calculated by the AI ​​model 212 is appropriate. In the judgment process 217, the timing at which the teacher detects an event in which intervention is required is compared with the intervention timing calculated by the AI ​​model 212. If the teacher does not detect an event in which intervention is required at the intervention timing calculated by the AI ​​model 212, it is judged that the AI ​​model 212 has erroneously detected the event in which intervention is required. Conversely, if the intervention timing has not been calculated by the AI ​​model 212 at the timing at which the teacher detects an event in which intervention is required, it is judged that the AI ​​model 212 has failed to detect the event in which intervention is required.

[0045] Feedback processing 218 is processing for feeding back the determination result of the intervention timing to the AI ​​model 212. In the feedback processing 218, a set of a false positive event that has been falsely detected by the AI ​​model 212 as an event requiring intervention, and video data in which the AI ​​model has detected the false positive event, is fed back to the AI ​​model 212. Also, in the feedback processing 218, a set of an undetected event that has not been detected by the AI ​​model 212 as an event requiring intervention, and video data in which the teacher has detected the undetected event is fed back to the AI ​​model 212.

[0046] 4. Other Applications of AI Video Analysis Systems The AI ​​video analysis system 10 can be applied to various uses other than the above Application Example 1 and Application Example 2. For example, it can be applied to monitoring workers in a factory or a logistics warehouse where the workers are working. In addition, the subject captured by the camera in the AI ​​video analysis system 10 is not limited to a person. The subject may be a vehicle on a road or a robot in a factory. According to the AI ​​video analysis system 10, a noteworthy event that occurs in the subject can be detected with high accuracy by the AI ​​model 12. [Explanation of symbols]

[0047] 10, 100, 200 AI video analysis system 11 Camera 12 AI models 13 people 14 Comparative Judgment Unit 20 Servers

Claims

1. An artificial intelligence (AI) video analysis system, a memory for storing a plurality of instructions; at least one processor communicatively coupled to the memory; The instructions are provided to the at least one processor: Obtaining an AI model trained to analyze video; Acquiring camera image data captured by a camera; Having the AI ​​model analyze the camera image data; acquiring a result of a human analysis of the camera image data; and providing feedback to the AI ​​model based on a comparison between an analysis result of the camera image data by the AI ​​model and an analysis result of the camera image data by the human. An AI video analysis system characterized by the above.

2. In the AI ​​video analysis system according to claim 1, Providing the feedback to the AI ​​model includes providing the AI ​​model with a set of undetected events that were detected in analysis by the human but not in analysis by the AI ​​model, and camera video data in which the human detected the undetected events. An AI video analysis system characterized by the above.

3. In the AI ​​video analysis system according to claim 1 or 2, Providing the feedback to the AI ​​model includes providing the AI ​​model with a set of false positive events that were detected by analysis by the AI ​​model but not by analysis by the human, and camera video data in which the AI ​​model detected the false positive events. An AI video analysis system characterized by the above.

4. A method for improving the accuracy of AI in an AI video analysis system, comprising: Obtaining an AI model trained to analyze video; Acquiring camera image data captured by a camera; Having the AI ​​model analyze the camera image data; acquiring a result of a human analysis of the camera image data; Providing feedback to the AI ​​model based on a comparison between an analysis result of the camera image data by the AI ​​model and an analysis result of the camera image data by the human. A method comprising:

5. A program including a plurality of instructions for execution by at least one processor, The instructions are provided to the at least one processor: Obtaining an AI model trained to analyze video; Acquiring camera image data captured by a camera; Having the AI ​​model analyze the camera image data; acquiring a result of a human analysis of the camera image data; and providing feedback to the AI ​​model based on a comparison between an analysis result of the camera image data by the AI ​​model and an analysis result of the camera image data by the human. A program characterized by:

Citation Information

Patent Citations

  • Self oil supply monitoring system and learning device

    JP2023007794A

  • Information processing device, information processing method, and program

    WO2020136761A1

  • Learning method and learning device for computer implementation

    JP2010518416A

  • Learning device, learning method, and program for learning device, as well as information output device, information output method, and information output program

    WO2019151411A1

  • Monitoring system, camera, analyzing device, and ai model generating method

    WO2022059123A1