Information processing apparatus

By combining the detection results of the first and second machine learning models in the information processing device, the false detection of object detection in the image is determined, and the problem of false detection of machine learning models under changes in image conditions is solved, and the simplified false detection judgment is achieved. General mechanism improves detection accuracy and robustness.

CN119027865APending Publication Date: 2024-11-26NOMURA RESEARCH INSTITUTE
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410644887.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-05-23
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Machine learning models are prone to false detection when detecting objects in images, especially when the shooting conditions change or image processing, resulting in increased false detection factors and complicated installation problems.

Method used

Design an information processing device to output object detection results using the first machine learning model in the detection image, and verify it through a second machine learning model with different characteristics from the predetermined ones, to determine whether the difference in the detection results meets the predetermined conditions to issue an error detection notification , realize a general false detection and discovery mechanism.

Benefits of technology

The common mechanism discovers false detection of machine learning models, which reduces the installation complexity caused by the increase in false detection factors, simplifies the false detection judgment process, and improves detection accuracy and robustness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027865A_ABST
    Figure CN119027865A_ABST
Patent Text Reader

Abstract

The information processing apparatus outputs a first detection result indicating a region in which an object is detected, among regions within the image, using a first machine learning model that detects the object in the image. The information processing apparatus also outputs a second detection result indicating a region in which the object is detected, among the regions within the image, using a second machine learning model whose predetermined characteristics related to object detection differ from the first machine learning model. Moreover, when the difference between the first detection result and the second detection result satisfies a predetermined condition, the information processing apparatus issues a notification that there is a false detection in the object detection of the first machine learning model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an information processing device for detecting an object in an image. Background Art

[0002] In recent years, the technology of detecting objects in images using models that have undergone machine learning such as deep learning (machine learning models) has become widely known. Although the accuracy of object detection by machine learning models is sometimes practical, false detections may occur due to the actual conditions of the image capture process (such as weather conditions) or if the image is intentionally processed.

[0003] In Japanese Patent Gazette No. 2022-131761, a technology is proposed that, when using a machine learning model to detect areas that may be people in images taken by a surveillance camera, determines whether the machine learning model mistakenly detects an area where insects are flying as an area that may be a person based on whether there is overexposure.

[0004] False detection of machine learning models may be caused by a variety of factors, such as changes in the situation when the image is taken. In Japanese Patent Gazette No. 2022-131761, a technology for detecting false detection of machine learning models caused by insect flight is proposed, but when assembled according to the factors of each false detection, the installation becomes more complicated as the false detection factors increase. Summary of the invention

[0005] The present invention is proposed in view of the above-mentioned problem, and its purpose is to discover the false detection of machine learning models with a universal mechanism.

[0006] In order to solve this problem, for example, the information processing device of the present invention is specifically as follows:

[0007] An information processing device for detecting an object in an image, comprising:

[0008] one or more processors;

[0009] A memory storing one or more instructions; when the one or more processors execute the one or more instructions, the information processing device:

[0010] outputting, using a first machine learning model for detecting objects in an image, a first detection result indicating an area in the image where the object is detected;

[0011] outputting a second detection result indicating a region in the image where the object is detected using a second machine learning model having a different predetermined characteristic related to object detection than the first machine learning model;

[0012] When the difference between the first detection result and the second detection result meets a predetermined condition, a notification is issued that there is a false detection in the object detection of the first machine learning model.

[0013] According to the present invention, false positives in machine learning models can be discovered through a general mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 FIG. 1 is a diagram schematically illustrating an information processing system according to an embodiment of the present invention.

[0015] Figure 2 A block diagram showing an example of the hardware configuration of the information processing device involved in this embodiment

[0016] Figure 3 is a block diagram showing an example of a functional configuration of an information processing device according to this embodiment

[0017] Figure 4 This is an explanatory diagram for explaining a case where it is determined that a detection model misdetection exists in this embodiment.

[0018] Figure 5 This is an explanatory diagram for explaining a situation where it is determined that there is no detection model misdetection in this embodiment.

[0019] Figure 6 is an explanatory diagram of the case where the false detection determination process involved in this embodiment is applied to a video

[0020] Figure 7 is an explanatory diagram of the case where the false detection determination process (using multiple verification models) involved in this embodiment is applied to a video

[0021] Figure 8 This is a flowchart for explaining a series of operations for performing false detection notification according to this embodiment.

[0022] Fig. 9 is a diagram showing an example of the data structure of the false detection notification information involved in this embodiment. DETAILED DESCRIPTION

[0023] Hereinafter, the embodiments are described in detail with reference to the accompanying drawings. In addition, the following embodiments are not limited to the inventions involved in the patent application, and the combination of features described in the embodiments is not necessarily required for the invention. Two or more of the multiple features described in the embodiments can be combined arbitrarily. In addition, the same reference number is marked for the same or similar structures, and repeated descriptions are omitted.

[0024] <Information Processing System Overview>

[0025] Reference Figure 1, an example of the information processing system involved in this embodiment is described. For example, the information processing system 10 includes an information processing device 100 and a communication device 104 for sending a captured image and receiving an object detection result. The device for sending the captured image and the device for receiving the object detection result can be different devices, for example, they can be respectively configured as an image sending device (such as a surveillance camera 102-1) for sending the captured image and a result receiving device (user terminal 102-2) for receiving the object detection result. The information processing system 10 may also include a service provider terminal 106. The information processing device 100 can communicate with the communication device 104 (or image sending device, result receiving device) and the service provider terminal 106 through a network. In addition, in this embodiment, although it is configured to include the information processing device 100 and the communication device 104 for sending the captured image and receiving the object detection result, it can also be configured so that the information processing device 100 works on the user's device.

[0026] For example, in order to implement a predetermined service including object detection, the information processing device 100 detects objects in a captured image using a machine learning model and sends an object detection result. In addition, in the following embodiments, the machine learning model executed to detect an object is referred to as a detection model. As long as the object detection function is included, the detection model can be various models. In this embodiment, the object may include a person, an animal, a plant, a vehicle, a building, daily necessities, food, etc. Therefore, the detection model may include, for example, a model for detecting a person in an image, a model for detecting an animal in an image, a model for detecting a specific item in an image (such as a vehicle or daily necessities, etc.), or a model for detecting multiple types of targets therein. The detection model may also be a model for detecting an object with specific features (such as a person performing a specific action, a person wearing a specific item, or a person of a specific gender, etc.). The object detection result may include information indicating the detected object (such as a box indicating the area of ​​the detected object, a color or text indicating the name of the object, or a numerical value such as a score for the object during detection).

[0027] The scheduled service may include, for example, detecting and labeling at least one of people, animals, plants, vehicles, buildings, daily necessities, foods, etc. from a captured static image or video, or providing information related to the detected target, or providing navigation services. It may also include providing services for analyzing information of static images or videos by detecting the above objects from static images or videos sent from fixed devices such as surveillance cameras. According to the implementation method of the service, the information processing device 100 may obtain images captured by an image sending device (such as a surveillance camera 102-1) and send the object detection results to a result receiving device (such as a user terminal 102-2 set up to monitor the images of the surveillance camera).

[0028] In addition to using the detection model to detect objects in the image, the information processing device 100 also performs the false detection determination processing described below. The false detection determination processing refers to determining whether there is a false detection in the object detection result of the object detection model, and if there is a false detection, taking a processing method to notify the existence of the false detection. For example, the information processing device 100 uses a machine learning model (also referred to as a verification model) in the false detection determination processing, and its predetermined characteristics for object detection are more robust than the object detection model. For example, for one or more characteristics (such as people, vehicles, or specific features, etc.), the verification model can detect more robustly than the detection model. Therefore, if the verification model detects an object, but the detection model does not detect the object, it is determined (due to some influence in the image) that there is a false detection in the detection result of the detection model. In addition, as a verification model, a model that can detect objects more robustly in an image containing adversarial sample attacks can be used. If the detection model fails to detect the target object in the image containing the adversarial sample, and the verification model detects the target object (or a part of the target object), it can be determined that there is a false detection in the detection result of the detection model. Thus, in this embodiment, there is no need to add a separate installation solution for the false detection factor, and the false detection of the machine learning model can be discovered through a general mechanism.

[0029] If the information processing device 100 determines that there is a false detection through the false detection determination process, it will notify the false detection. Figure 1 In the example shown, for example, the false detection notification is sent to the service provider terminal 106. The service provider terminal 106 is a terminal managed by the service provider. For example, the service provider provides an object detection service for detecting objects in the image captured by the surveillance camera 102-1. The service provider terminal 106 can perform, for example, hyperparameter setting of the detection model, learning and deployment of the detection model, and confirmation of the running status of the detection model on the detection model. The information processing device 100 sends the false detection notification to the service provider terminal 106, so that the service provider can know the existence of false detection in the detection model it uses. The service provider can prepare the necessary learning data to let the detection model learn again according to the content of the false detection, or replace it with another machine learning model with better required characteristics. In this embodiment, although the information processing device 100 sends the false detection notification to the service provider terminal 106 as an example, the information processing device 100 can also send the false detection notification to the user terminal 102-2 or the communication device 104 according to the content of the service.

[0030] The information processing device 100 is, for example, a server device. The information processing device 100 may be, for example, a server device that implements a cloud service platform, and the object detection process and the false detection determination process may also be implemented on the cloud service platform. However, the information processing device 100 may be an edge node configured on the network or a node constituting a P2P network. Alternatively, the information processing device 100 may also be a virtual machine configured on the cloud service platform.

[0031] In this embodiment, the case where object detection processing using a detection model and false positive determination processing using a verification model are performed on the information processing device 100 (i.e., the server device) is used as an example for explanation. However, this embodiment can also be applied to the case where the processing is performed by multiple server devices implementing a cloud service platform. In addition, this embodiment is also applicable to the case where object detection processing and false positive determination processing are performed on a server device such as a server device. Figure 1 The case where the communication device 104 is shown, or the ....

[0032] The communication device 104 may be, for example, a tablet device or a smart phone, or a personal computer, etc. The service provider terminal 106 may be, for example, a personal computer, or a tablet device or a smart phone, etc.

[0033] <Hardware Configuration Example of Information Processing Device>

[0034] Reference Figure 2 , an example of the hardware configuration of the information processing device 100 is described. The information processing device 100 includes a memory 202, a processor 204, a communication interface 206, a storage 208, an input interface 210, and a power supply 212. These elements are respectively connected to a bus 214, and communicate with each other via the bus 214.

[0035] The memory 202 is a volatile storage medium such as DRAM, which is used to temporarily store data and programs. In addition, the memory 208 is a non-volatile storage medium that permanently stores data and programs. The stored program includes one or more instructions that can be executed by the processor. The memory 208 can be, for example, a semiconductor memory or a hard disk. The memory 208 can store various data such as learning data for neural network learning, test data for testing a neural network that has completed learning, captured images received from the monitoring camera 102-1 or the communication device 104, and false detection notification data described later.

[0036] The processor 204 includes, for example, a computing circuit such as a central processing unit (CPU). The processor 204 may be composed of one or more processors. The processor 204 may further include a computing circuit (such as a GPU) or dedicated hardware for quickly performing statistical processing such as machine learning, and may also include a memory. The processor 204 implements various functions of the information processing device 100 by expanding and executing the program stored in the memory 208 in the memory 202.

[0037] The communication interface 206 is an interface for transmitting and receiving data with an external device of the information processing device 100. The communication interface 206 may include a communication circuit capable of communicating in a communication manner in accordance with various standards. The communication interface 206 is connected to a network and communicates with the external device via the network. Figure 1 The information processing apparatus 100 exchanges data with the communication apparatus 104 and the like. The input interface 210 is a device for receiving input from, for example, an administrator of the information processing apparatus 100, but the input interface 210 may not be provided.

[0038] The power supply 212 is a circuit or module for supplying electric power to ensure the operation of each unit of the information processing device 100. The power supply 212 may be configured to include a battery.

[0039] <Functional Configuration Example of Information Processing Device>

[0040] Next, refer to Figure 3 , an example of the functional configuration of the information processing device 100 is described. For example, Figure 3 The functional configuration example shown can be implemented, for example, by the processor 204 expanding in the memory 202 and executing a program stored in the storage 208. In addition, the various functional blocks described in this embodiment can be integrated or separated, and the described functions can also be implemented by other blocks. In addition, the content described as hardware can be implemented by software, and vice versa.

[0041] The data acquisition unit 310 can, for example, obtain captured images from an external device. The captured images can also be included in multiple frames constituting a video. For example, the data acquisition unit 310 can obtain a video consisting of multiple consecutive frames of images from the monitoring camera 102-1 or the communication device 104. In addition, the data acquisition unit 310 can also obtain captured images received by the memory 208 and temporarily or permanently stored from the communication device 104, etc. In addition, the captured images stored in the external memory of the information processing device 100 can also be obtained.

[0042] The detection model processing unit 312 includes a detection model execution unit 314 and a detection result output unit 316 , and performs object detection processing through these functional configurations.

[0043] The detection model execution unit 314 inputs the image (captured image) acquired by the data acquisition unit 310, executes the learned detection model, and detects the object in the captured image. As described above, the detection model can be various models, as long as it is a model that includes an object detection function. For example, the detection model can adopt a detection model that can detect multiple types of targets. The detection model execution unit 314 can adopt a well-known model called YOLO or FasterR-CNN as a detection model. The detection model outputs, for example, a box indicating the area of ​​the detected object, a color or text indicating the name of the detected object, a numerical value indicating the probability of detection, etc.

[0044] The detection result output unit 316 outputs information related to the object detected by the detection model (i.e., the object detection result). The detection result output unit 316 uses the output of the detection model execution unit 314 to superimpose a frame indicating the object area on the captured image to generate an object detection result. The detection result output unit 316 can, for example, send the generated object detection result to the communication device 104 or the user terminal 102-2.

[0045] The verification model processing section 318 includes a verification setting section 320 , a verification model execution section 322 , a false detection determination section 324 , and a notification section 326 , and the verification model processing section 318 performs false detection determination processing through these functional configurations.

[0046] The verification setting unit 320 sets the verification model for the false detection determination process according to the instruction information from the service provider terminal 106, for example. The instruction information includes information about the designation and use of the verification model. The instruction information includes, for example, identification information, version information, execution frequency information, etc. of the verification model used for verification. The identification information of the verification model is, for example, information for identifying the model that has completed learning and is stored in the model data 330. Multiple verification models can be specified by specifying multiple identification information.

[0047] The verification setting unit 320, for example, sets a verification model that can detect a specific object more robustly (than the detection model) and is specifically used to detect the specific object based on the identification information of the verification model. If a detection model that can detect multiple types of objects is used, the identification information of multiple verification models can be specified. In this case, a model that can detect people more robustly (than the verification model) and a model that can detect vehicles more robustly can be specified. Alternatively, when a detection model that can detect multiple types of objects is used, the verification setting unit 320 can set a verification model that can detect multiple types of objects like the detection model, but can detect each object more robustly, based on the identification information of the verification model. In this way, in this embodiment, the required verification model can be set according to the detection model used for object detection processing. This means that a general mechanism can be used to discover false detections of machine learning models that may be caused by various factors.

[0048] The version information in the indication information indicates the version of the verification model. For example, the verification model that has been relearned using newly added learning data may be specified.

[0049] The execution frequency in the indication information, for example, specifies how many frames in a plurality of consecutive frames in a video the verification model is to be executed on. Although the verification model can detect objects more robustly than the detection model, its execution speed may be slower than the detection model. Therefore, when executing the detection model on a plurality of consecutive frames, the verification model can be set to execute at a lower frequency. For example, for an image of a plurality of frames, the detection model outputs detection results for the first number of images per second, while the verification model outputs detection results for the second number (less than the first number) of images per second in the plurality of frames. The execution frequency can be expressed, for example, by the ratio of the number of executions of the detection model to the number of executions of the verification model. In this way, in the information processing device 100, the processing load of the information processing device 100 can be reduced or the operation of the information processing device 100 can be accelerated. In addition, the use of a detection model that runs faster can achieve fast detection services, while also detecting false detections of the detection model.

[0050] The verification model execution unit 322 inputs the image (photographed image) acquired by the data acquisition unit 310 according to the setting of the verification setting unit 320, and executes the above-mentioned verification model. The verification model will output, for example, a frame indicating the area of ​​the detected object. However, in the present embodiment, it is not limited to a frame indicating the area of ​​the object, and it can also be other forms such as information indicating the shape of the area of ​​the object. The verification model can also output information corresponding to the frame indicating the name of the detected object.

[0051] The false detection determination unit 324 determines whether there is a false detection in the detection result of the detection model based on the detection result of the detection model and the detection result of the verification model. For example, when the difference between the detection result of the detection model and the detection result of the verification model is large enough to meet a certain condition, the false detection determination unit 324 determines that there is a false detection in the detection result of the detection model.

[0052] Reference Figure 4 The case where it is determined that the detection model has a false detection is described. The captured image 400 is a captured image acquired by the data acquisition unit 310. The captured image 400 is input to the detection model 402 and the verification model 404 respectively. The detection model 402 and the verification model 404 respectively detect objects in the captured image 400. In this example, the verification model 404 is a model that detects a person more robustly than the detection model 402.

[0053] Detection result 406 schematically shows an example of an object detected by detection model 402. Boxes 408 and 410 respectively indicate the regions of the object detected by detection model 402. For ease of understanding, detection result 406 superimposes boxes 408 and 410 on captured image 400. In this example, detection model 402 detects a person in the box 408 region and a bicycle in the box 410 region. Detection region 420 schematically shows what it would look like if only regions 422 and 424 of the object detected by detection model 402 were extracted.

[0054] On the other hand, the detection result 412 schematically shows an example of an object detected by the verification model 404. Boxes 414 and 416 respectively indicate the regions of the object detected by the verification model 404. For ease of understanding, the detection result 412 superimposes the boxes 414 and 416 on the captured image 400. In this example, the verification model 404 detects a person in the box 414 region and a person in the box 416 region. The mapping image 426 is a binary image in which the regions 428 and 430 of the object detected by the verification model 404 are assigned a value of "1" and the other regions are assigned a value of "0".

[0055] The false detection determination unit 324 extracts the difference between the mapping image 426 and the detection area 420. For example, the difference mapping image 432 is generated by assigning a value of "1" to the object area of ​​the mapping image 426 and the area (non-overlapping area) where there is a difference between the object areas 422 and 424. In the difference mapping image 432, an area (difference area 434) is generated according to the difference between the area of ​​the object detected by the detection model 402 and the area of ​​the object detected by the verification model 404. If the false detection determination unit 324, for example, determines that the size of the difference area 434 is above a predetermined threshold, it is determined that there is a false detection in the detection result of the detection model 402. In this embodiment, a method of generating a difference mapping to determine the existence of a false detection is adopted. In addition to this, a method of only comparing the number of object detections of the target image, a method of comparing the number of object detections of the target image and the respective coordinate positions (such as the centroid coordinates of each object), a method of comparing the number of object detections of the target image and the respective coordinate positions and areas, etc. can also be adopted.

[0056] Next, refer to Figure 5 The case where it is determined that there is no false detection of the detection model is described below. The captured image 500 is the captured image acquired by the data acquisition unit 310. Figure 4 The captured image 400 is the same as shown. The captured image 500 is input to the detection model 502 and the verification model 504 respectively. The detection model 502 and the verification model 504 respectively detect objects in the captured image 500. For example, the detection model 502 is a model for detecting people in the image, and the verification model 504 is a model that detects people more robustly than the detection model 502.

[0057] Detection result 506 schematically shows an example of an object detected by detection model 502. Boxes 508 and 510 respectively indicate the regions of the object detected by detection model 502. For ease of understanding, detection result 506 superimposes boxes 508 and 510 on captured image 500. In this example, detection model 502 detects a person in the box 508 region and a person in the box 510 region. Detection region 520 schematically shows what it would look like if only regions 522 and 524 of the object detected by detection model 502 were extracted.

[0058] On the other hand, the detection result 512 schematically shows an example of an object detected by the verification model 504. Boxes 514 and 516 respectively indicate the regions of the object detected by the verification model 504. For ease of understanding, the detection result 512 superimposes the boxes 514 and 516 on the captured image 500. In this example, the verification model 504 detects a person in the box 514 region and a person in the box 516 region. The mapping image 526 is a binary image in which the value "1" is assigned to the regions 528 and 530 of the object detected by the verification model 504 and the value "0" is assigned to the other regions.

[0059] The false detection determination unit 324 extracts the difference between the mapping image 526 and the detection area 520. For example, a difference mapping image 532 is generated by assigning a value of "1" to the object area of ​​the mapping image 526 and the area (non-overlapping area) where the difference exists between the object areas 522 and 524. In the difference mapping image 532, there is no difference between the area of ​​the object detected by the detection model 502 and the area of ​​the object detected by the verification model 504. Therefore, for example, since there is no difference area with a size greater than a predetermined threshold, the false detection determination unit 324 determines that there is no false detection in the detection result of the detection model 502. That is, if an object is not detected in an area in the detection result of the detection model, but an object with an area greater than a predetermined size is detected in the detection result of the verification model, it is determined that there is a false detection.

[0060] The determination made by the false detection determination unit 324 can also be applied to the case where multiple verification models are used. For example, the false detection determination unit 324 generates a mapping image that is a synthesis of mapping images 526 (OR in the area of ​​"1"), and the mapping images 526 come from a first verification model that detects people more robustly than the detection model and a second verification model that detects vehicles more robustly than the detection model. A differential mapping image is then generated between the synthesized mapping image and object areas 422, 522, etc. That is, if an object is detected in an area where the object is not detected in the detection results of the detection model, but an object of an area larger than a predetermined size is detected in at least one of the detection results of the first verification model and the detection results of the second verification model, it can be determined that a false detection exists. In this way, false detections related to multiple characteristics can be determined by a simple mechanism. This means that there is no need to introduce complex implementation schemes for each factor that causes false detection, and the operation of the information processing device is simpler and more stable. In addition, as a method of using the first verification model and the second verification model in combination, in addition to the above, it can also be a structure in which the second verification model is used simultaneously based on the detection results of the first verification model. For example, when the detection reliability of each object (i.e., the detection result of the first verification model) is equal to or lower than a predetermined threshold, the second verification model also performs object detection, and as described above, the respective detection results are synthesized.

[0061] Refer again Figure 3 When the false detection determination unit 324 determines that a false detection exists, the notification unit 326 sends a false detection notification to an external device (such as the service provider terminal 106). The notification unit 326 may send a false detection notification every time a false detection occurs. Alternatively, the notification unit 326 may also create a notification information recording related to the false detection notification, and send the generated notification information to the external device at a predetermined time interval.

[0062] Notification information can be, for example, Fig. 9The data structure 900 shown in FIG. 900 may be composed of, for example, a frame number, content of false detection, identification information of a detection model, identification information of a verification model, and identification information of a video.

[0063] The frame number is the number of the frame in the video that is determined to have a false detection. The frame is the frame number of the video identified by the "video identification information" of the data structure 900. The content of the false detection is some indicative information, such as which characteristics indicate that a false detection has occurred. The notification unit 326 can describe the characteristics based on the detection result of the verification model when the false detection occurs. For example, the notification unit 326 can use the detection result of the verification model corresponding to the area (differential area 434) determined to have a false detection (the name of the object involved in box 416, here is a person) to set the content of the false detection to "human missed detection." The identification information of the detection model and the identification information of the verification model are respectively the identification information of the model used when it is determined that a false detection exists.

[0064] By receiving the notification information generated by the notification unit 326 through the service provider terminal 106, the service provider can easily understand what kind of false detection has occurred in the running detection model and in what kind of image the false detection has occurred. Therefore, the service provider can easily improve the service quality using the detection model by reading the notification information, or determining and supplementing the learning data used when re-learning the detection model, or replacing the detection model.

[0065] The model data 330 is a database storing the data of the detection model and the verification model. The model data 330 includes information such as the hyper parameters and learning completion parameters of each model.

[0066] The verification setting data 332 stores the various information set by the verification setting unit 320, such as identification information of a verification model used for verification, version information of the verification model, execution frequency information of executing the verification model, and the like.

[0067]

[0068] Next, refer to Figure 8 A series of actions for performing false detection notification in the information processing device 100 are described. This process is implemented by the processor 204 executing the computer program recorded in the memory 208. In addition, in the following description, the execution of the detection model and the execution of the verification model are described as an example with the same frequency (i.e., for all frames), but as mentioned above, the execution frequency of the verification model can be lower than the execution frequency of the detection model. That is, Figure 6 As shown, the detection model 600 is executed on each frame in the video (frame 610, etc.), and correspondingly, the verification model 602 can be executed at a lower frequency (ie, for frames 610, 614, 616).

[0069] In S801, the data acquisition unit 310 acquires an image (photographed image) from the memory 208 or an external device. In S802, the detection model processing unit 312 performs object detection processing. For example, the detection model execution unit 314 executes the detection model on the acquired image to detect an object in the image.

[0070] Next, the verification model processing unit 318 performs false detection determination processing. Specifically, in S803, the verification model execution unit 322 executes the verification model on the acquired image to detect the object in the image. Then, in S804, the false detection determination unit 324 generates a mapping image (such as the mapping image 426) indicating the area of ​​the object detected by the verification model. In S805, the false detection determination unit 324 removes the area detected by the detection model from the area of ​​the object in the mapping image (i.e., performs false detection determination). Figure 4 The difference extraction described in the above generates the difference mapping image 432). In S806, the false detection determination unit 324 calculates the total area of ​​the region (difference region 434) in the remaining mapping image, and in S806, determines whether the area is above the threshold. If the false detection determination unit 324 determines that the area is above the threshold, the processing flow enters S808, otherwise the processing flow returns to S801.

[0071] In S808, the notification unit 326 generates notification information related to the false detection, and notifies the external device of the existence of the false detection in real time or at a predetermined time. In S809, the detection model processing unit 312 determines whether all images related to the video have been processed. If it is determined that all images have been processed, the processing flow ends, otherwise the processing flow returns to S801.

[0072] The above actions can also be applied to situations where multiple verification models are used. Figure 7 As shown, the first verification model 702 and the second verification model 704 in the plurality of verification models can be run alternately. For example, the first verification model 702 is executed for frames 610 and 616, and the second verification model 704 is executed for frame 614. At this time, the execution frequency of the first verification model 702 and the second verification model 704 added together is lower than the execution frequency of the detection model 600.

[0073] As shown above, in the present embodiment, the detection model execution unit 314 is configured to use the detection model of the object in the detection image to output a first detection result indicating the area in the image where the object is detected. And the verification model execution unit 322 is configured to use a verification model having object detection characteristics different from the detection model to output a second detection result indicating the area in the image where the object is detected. In addition, the false detection determination unit 324 is configured to determine whether the difference between the two detection results is large enough to meet a predetermined condition, and if the difference is large enough to meet the condition, determine that there is a false detection in the object detection of the detection model. In the event that there is a false detection of the detection model, the notification unit 326 sends a notification message to the service provider terminal 106, for example. In this way, in the present embodiment, there is no need to add an installation plan based on the factors that cause the false detection, and the false detection of the machine learning model can be discovered through a general mechanism.

[0074] The present invention is not limited to the above-described embodiment, and various modifications and changes can be made within the scope of the gist of the invention.

Claims

1. An information processing device for detecting an object in an image, include: one or more processors; A memory storing one or more instructions; When the one or more processors execute the one or more instructions, the information processing device: outputting, using a first machine learning model for detecting objects in an image, a first detection result indicating an area in the image where the object is detected; outputting a second detection result indicating a region in the image where the object is detected using a second machine learning model having a different predetermined characteristic related to object detection than the first machine learning model; When the difference between the first detection result and the second detection result meets a predetermined condition, a notification is issued that there is a false detection in the object detection of the first machine learning model.

2. An information processing device according to claim 1, wherein the second machine learning model is more robust than the first machine learning model with respect to predetermined characteristics of object detection.

3. The information processing device according to claim 2, in, The notification includes: if an object having an area larger than a predetermined size is detected in an area where no object is detected in the first detection result, in the second detection result, it is determined that the predetermined condition is satisfied.

4. The information processing device of claim 1, wherein the one or more instructions further cause the information processing device to: Acquire multiple consecutive frames of images; The first machine learning model outputs the first detection result for a first number of images per second in the plurality of frames of images; The second machine learning model outputs the second detection result for a second number (less than the first number) of images per second among the multiple frames of images.

5. The information processing device of claim 3, wherein the one or more instructions further cause the information processing device to: outputting a third detection result indicating a region in the image where the object is detected using a third machine learning model that is more robust than the first machine learning model using a second predetermined characteristic related to object detection; Said notification include: If an object is not detected in an area in the first detection result, but an object in an area larger than the predetermined size is detected in at least one of the second detection result and the third detection result, a notification is issued that there is a false detection in the object detection of the first machine learning model. 6 . The information processing apparatus according to claim 5 , wherein the notification includes notifying information on the object involved in the misdetection.

7. According to the information processing device according to claim 1, the second machine learning model has the characteristic of detecting objects of a specific category more robustly than the first machine learning model.

Citation Information

Patent Citations

  • Sample labeling method and computer storage medium

    CN110135225A

  • Model training method and device, electronic equipment, storage medium and product

    CN114445682A

  • Multi-model detection of objects

    EP3772018A1

  • Site-Based Calibration of Object Detection Parameters

    US20220414382A1