Information processing device

The information processing device uses a verification model to identify and correct false detections in object detection models by comparing results, addressing the issue of false positives in machine learning models through a general-purpose mechanism.

JP2024170312AActive Publication Date: 2024-12-06NOMURA RESEARCH INSTITUTE

Patent Information

Application Number
JP2024084230
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-05-23
Publication Date
2024-12-06
Estimated Expiration
2044-05-23

AI Technical Summary

Technical Problem

Machine learning models used for object detection in images are prone to false detections due to various factors, including changes in image-taking situations and adversarial attacks, complicating their implementation.

Method used

An information processing device employs a first machine learning model for object detection and a second verification model with different characteristics to identify false positives by comparing detection results, notifying false detections when a predetermined condition is met.

Benefits of technology

This approach allows for the detection of false positives in machine learning models using a general-purpose mechanism, simplifying the process and enabling effective identification and correction of false detections.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024170312000001_ABST
    Figure 2024170312000001_ABST
Patent Text Reader

Abstract

To achieve a technology capable of discovering an erroneous detection a machine learning model with a general purpose mechanism.SOLUTION: An information processing device outputs a first detection result showing an area in which an object is detected among areas in an image by using a first machine learning model for detecting the object in the image. Also, the information processing device outputs a second detection result showing an area in which the object is detected among the areas in an image by using a second machine learning model whose prescribed characteristic about the detection of the object is different from that of the first machine learning model. Further, the information processing device notifies an external device or the like of the existence of an erroneous detection about the detection of the object by the first machine learning mode in the case that difference between the first detection result and the second detection result satisfies a prescribed condition.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device that detects an object in an image. [Background technology]

[0002] In recent years, a technology has become known that uses a model trained by machine learning such as deep learning to detect objects in an image (machine learning model). Although the accuracy of object detection using machine learning models can be practical in some cases, false detections may occur depending on the circumstances when the image is taken (e.g., weather, etc.) or if the image has been intentionally altered.

[0003] Patent Document 1 proposes a technology that, when a machine learning model is used to detect areas that resemble humans in an image captured by a surveillance camera, determines whether the machine learning model has erroneously detected an area where an insect is flying as a human-like area by checking whether there is blown-out highlights. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-131761 A Summary of the Invention [Problem to be solved by the invention]

[0005] False positives caused by machine learning models can occur due to various factors such as changes in circumstances when an image is captured. Patent Document 1 proposes a technology for detecting false positives caused by flying insects in machine learning models, but when implementing the technology according to the individual factors of false positives, there is a problem that the implementation becomes more complicated as the factors of false positives increase.

[0006] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that can detect false positives in machine learning models using a general-purpose mechanism. [Means for solving the problem]

[0007] In order to solve this problem, for example, the information processing device of the present invention comprises: An information processing device for detecting an object in an image, one or more processors; and a memory storing one or more instructions, the one or more instructions, when executed by the one or more processors, causing the information processing device to: outputting a first detection result indicating an area in the image in which the object is detected using a first machine learning model for detecting an object in the image; outputting a second detection result indicating an area within the image in which an object has been detected using a second machine learning model having a predetermined characteristic related to object detection different from that of the first machine learning model; and When a difference between the first detection result and the second detection result satisfies a predetermined condition, the presence of a false positive in the detection of the object by the first machine learning model is notified. Effect of the Invention

[0008] According to the present invention, it is possible to discover false positives in machine learning models using a general-purpose mechanism. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing device according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device according to an embodiment of the present invention; [Figure 4] FIG. 1 is a diagram for explaining a case in which it is determined that a false positive occurs due to a detection model in this embodiment. [Diagram 5] FIG. 1 is a diagram for explaining a case in which it is determined that there is no erroneous detection by a detection model in the present embodiment. [Figure 6]FIG. 1 is a diagram illustrating a case where the erroneous detection determination process according to the present embodiment is applied to a moving image. [Figure 7] FIG. 1 is a diagram for explaining a case where the false positive determination process (using multiple verification models) according to the present embodiment is applied to a moving image. [Figure 8] A flowchart for explaining a series of operations for notifying an erroneous detection according to the present embodiment. [Figure 9] FIG. 13 is a diagram showing an example of the data configuration of notification information of an erroneous detection according to the embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] <Outline of information processing system> An example of an information processing system according to the present embodiment will be described with reference to FIG. 1. The information processing system 10 includes, for example, an information processing device 100 and a communication device 104 that transmits a captured image and receives an object detection result. The device that transmits the captured image and the device that receives the object detection result may be separate devices, and may be configured separately, for example, as an image transmitting device (for example, a surveillance camera 102-1) that transmits the captured image and a result receiving device (user terminal 102-2) that receives the object detection result. The information processing system 10 may also include a service provider terminal 106. The information processing device 100 can communicate with the communication device 104 (or the image transmitting device or the result receiving device) and the service provider terminal 106 through a network. Note that, in the present embodiment, the information processing system 10 includes the information processing device 100 and the communication device 104 that transmits a captured image and receives an object detection result, but the information processing device 100 may be operated on a user's device.

[0012] For example, the information processing device 100 detects an object in a captured image using a machine learning model as an implementation of a predetermined service including object detection, and transmits the object detection result. In the following embodiment, a machine learning model executed to detect an object is referred to as a detection model. The detection model may be various models including an object detection function. In this embodiment, the object may include, for example, a person, an animal, a plant, a vehicle, a building, daily necessities, food, etc. Therefore, the detection model may include, for example, a model that detects a person in an image, a model that detects an animal in an image, a model that detects a specific item (for example, a vehicle or daily necessities, etc.) in an image, or a model that detects a plurality of types of objects. In addition, the detection model may be a model that detects an object having a specific characteristic (for example, a person performing a specific action, a person wearing a specific item, or a person of a specific gender, etc.). The object detection result may include information indicating the detected object (for example, a frame indicating the area of ​​the detected object, a color or text indicating the name of the object, or a numerical value such as a score for the object at the time of detection).

[0013] The predetermined service may include, for example, a service for detecting at least one of people, animals, plants, vehicles, buildings, daily necessities, foodstuffs, etc. from a captured still image or video and annotating the detected object, providing information related to the detected object, or providing navigation. The predetermined service may also include a service for providing information for analyzing a still image or video by detecting the above-mentioned object from the still image or video transmitted from a fixed device such as a surveillance camera. Depending on the implementation of the service, the information processing device 100 may obtain an image captured from an image transmitting device (e.g., a surveillance camera 102-1) and transmit an object detection result to a result receiving device (e.g., a user terminal 102-2 installed to monitor the image of the surveillance camera).

[0014] Further, the information processing device 100 executes a false detection determination process described later in addition to detecting an object in an image using a detection model. The false detection determination process is a process for determining whether a false detection exists in a detection result of an object by an object detection model, and notifying the existence of the false detection if a false detection exists. In the false detection determination process, the information processing device 100 uses, for example, a machine learning model (also simply referred to as a verification model) in which a predetermined characteristic related to object detection is more robust than an object detection model. For example, the verification model can detect one or more characteristics, such as a person, a vehicle, or a specific feature, more robustly than the detection model. For this reason, if an object is detected by the verification model while the object is not detected by the detection model, it is considered that a false detection exists in the detection result of the detection model (due to some influence in the image). Furthermore, as the verification model, a model capable of robustly detecting an object in an image including an attack by an adversarial sample can be used. If the detection model cannot detect the target object in an image including an adversarial sample, while the verification model can detect the target object (or a part thereof), it can be determined that a false detection exists in the detection result by the detection model. In this way, in this embodiment, instead of adding individual implementations according to the causes of false positives, it is possible to detect false positives in machine learning models using a general-purpose mechanism.

[0015] When the information processing device 100 determines that a false detection exists by the false detection determination process, it notifies the false detection. In the example shown in FIG. 1, the false detection notification is transmitted to, for example, the service provider terminal 106. For example, the service provider terminal 106 is a terminal managed by the service provider, and the service provider provides an object detection service for detecting an object in an image captured by the surveillance camera 102-1. The service provider terminal 106 can perform, for example, setting hyperparameters of the detection model, learning and deploying the detection model, and checking the operation status of the detection model, with respect to the detection model. The information processing device 100 transmits a false detection notification to the service provider terminal 106, so that the service provider can grasp the existence of a false detection in the detection model being used. Depending on the content of the false detection, the service provider can prepare necessary learning data to retrain the detection model, or replace it with another machine learning model having superior required characteristics. In this embodiment, an example is described in which the information processing device 100 transmits a notification of a false positive to the service provider terminal 106. However, depending on the content of the service, the information processing device 100 may transmit a notification of a false positive to the user terminal 102-2 or the communication device 104.

[0016] The information processing device 100 is, for example, a server device. The information processing device 100 may be, for example, a server device that realizes a platform for a cloud service, and the above-mentioned object detection process and erroneous detection determination process may be realized on the platform for the cloud service. However, the information processing device 100 may be an edge node disposed on a network, or a node constituting a P2P network. Alternatively, the information processing device 100 may be a virtual machine configured on the platform for the cloud service.

[0017] In addition, in this embodiment, a case will be described in which the object detection process using the detection model and the erroneous detection determination process using the verification model are executed on the information processing device 100, which is a server device. However, this embodiment is also applicable to a case in which the process is executed by a plurality of server devices that realize a cloud service platform. This embodiment is also applicable to a case in which the object detection process and the erroneous detection determination process are executed in the communication device 104 shown in FIG. 1, or in the surveillance camera 102-1 or the user terminal 102-2.

[0018] The communication device 104 is, for example, a tablet device or a smartphone, but may also be a personal computer, etc. Furthermore, the service provider terminal 106 is, for example, a personal computer, but may also be a tablet device or a smartphone.

[0019] <Example of hardware configuration of information processing device> 2, an example of a hardware configuration of the information processing device 100 will be described. The information processing device 100 includes a memory 202, a processor 204, a communication interface 206, a storage 208, an input interface 210, and a power supply 212. Each of these elements is connected to a bus 214 and communicates with each other via the bus 214.

[0020] The memory 202 is a volatile storage medium such as a DRAM, and temporarily stores data and programs. The storage 208 is a non-volatile storage medium that permanently stores data and programs. The stored programs include one or more instructions that can be executed by a processor. The storage 208 may be, for example, a semiconductor memory or a hard disk. The storage 208 can store various types of data, such as training data for training a neural network, test data for testing a trained neural network, captured images received from the surveillance camera 102-1 or the communication device 104, and notification data of false detections, which will be described later.

[0021] The processor 204 includes, for example, an arithmetic circuit such as a central processing unit (CPU). The processor 204 may be configured with one or more processors. The processor 204 may further include an arithmetic circuit (for example, a GPU) or dedicated hardware for executing statistical processing such as machine learning at higher speed, and may include an internal memory. The processor 204 realizes various functions of the information processing device 100 by expanding a program stored in the storage 208 into the memory 202 and executing it.

[0022] The communication interface 206 is an interface for transmitting and receiving data between the information processing device 100 and an external device. The communication interface 206 may include a communication circuit capable of communicating in a communication method conforming to various standards. The communication interface 206 is connected to a network, and exchanges data with the communication device 104 and the like shown in Fig. 1 via the network. The input interface 210 is, for example, a device for accepting input from an administrator of the information processing device 100, but may not be required.

[0023] The power supply 212 is a circuit or module for providing power for operating each unit of the information processing device 100. The power supply 212 may be configured to include a battery.

[0024] <Example of functional configuration of information processing device> Next, a functional configuration example of the information processing device 100 will be described with reference to Fig. 3. The functional configuration example shown in Fig. 3 can be realized, for example, by the processor 204 expanding a program stored in the storage 208 into the memory 202 and executing it. Each of the functional blocks described in this embodiment may be integrated or separated, and the functions described may be realized by another block. Also, what is described as hardware may be realized by software, and vice versa.

[0025] The data acquisition unit 310 acquires a captured image from, for example, an external device. The captured image may be included in a plurality of frames constituting a video. For example, the data acquisition unit 310 may acquire a video composed of a plurality of consecutive frame images from the surveillance camera 102-1 or the communication device 104. The data acquisition unit 310 may also acquire a captured image received from the communication device 104 or the like and temporarily or permanently stored in the storage 208. Furthermore, the data acquisition unit 310 may acquire a captured image stored in a storage external to the information processing device 100.

[0026] The detection model processing unit 312 includes a detection model execution unit 314 and a detection result output unit 316, and executes object detection processing using these functional components.

[0027] The detection model execution unit 314 inputs an image (photographed image) acquired by the data acquisition unit 310, executes the learned detection model, and detects an object in the photographed image. As described above, the detection model may be various models as long as it includes an object detection function. For example, a detection model capable of detecting multiple types of objects can be used as the detection model. The detection model execution unit 314 can use a known model called, for example, YOLO or Faster R-CNN as the detection model. The detection model outputs, for example, a frame indicating the area of ​​the detected object, a color or text indicating the name of the detected object, and a numerical value indicating the probability of detection.

[0028] The detection result output unit 316 outputs information about the object detected by the detection model (i.e., the object detection result). The detection result output unit 316 uses the output from the detection model execution unit 314 to superimpose a frame indicating the area of ​​the object on the captured image to generate the object detection result. The detection result output unit 316 can transmit the generated object detection result to, for example, the communication device 104 or the user terminal 102-2.

[0029] The verification model processing unit 318 includes a verification setting unit 320, a verification model execution unit 322, an erroneous detection determination unit 324, and a notification unit 326, and the verification model processing unit 318 executes the erroneous detection determination process using these functional components.

[0030] The verification setting unit 320 sets a verification model to be used in the false detection determination process, for example, according to instruction information from the service provider terminal 106. The instruction information includes information regarding the specification of the verification model and the usage mode. The instruction information includes, for example, identification information of the verification model to be used for verification, version information, and execution frequency information. The identification information of the verification model is, for example, information for identifying a trained model stored in the model data 330. It is also possible to specify multiple identification information to specify multiple verification models.

[0031] The verification setting unit 320 sets a verification model specialized for detecting a specific object, which can detect the specific object more robustly (than the detection model) by, for example, the identification information of the verification model. When a detection model capable of detecting multiple types of objects is used, the identification information of multiple verification models may be specified. In this case, a model capable of detecting a person more robustly (than the verification model) and a model capable of detecting a vehicle more robustly can be specified. Alternatively, when one detection model capable of detecting multiple types of objects is used, the verification setting unit 320 can set one verification model capable of detecting multiple types of objects similar to the detection model, but more robustly detect each object, by the identification information of the verification model. In this way, in this embodiment, a desired verification model can be set according to the detection model used in the object detection process. That is, it becomes possible to discover erroneous detections of machine learning models that may occur due to various factors using a more versatile mechanism.

[0032] The version information in the instruction information indicates the version of the validation model, and can specify, for example, the validation model after re-learning using the added training data.

[0033] The execution frequency in the instruction information specifies, for example, how many frames out of a plurality of consecutive frames in a video the verification model is to be executed for. The verification model may detect an object more robustly than the detection model, but may have a slower execution speed than the detection model. For this reason, when the detection model is executed for a plurality of consecutive frames, it can be set to execute the verification model less frequently. For example, the detection model outputs detection results for a first number of images out of the images of the plurality of frames per second, and the verification model outputs detection results for a second number (a number smaller than the first number) out of the images of the plurality of frames per second. The execution frequency may be expressed, for example, as a ratio of the number of executions of the detection model to the number of executions of the verification model. In this way, the information processing device 100 can reduce the processing load of the information processing device 100 and speed up the operation of the information processing device 100. In addition, a high-speed detection service can be realized by using a detection model that operates at a higher speed, and at the same time, erroneous detection by the detection model can be detected.

[0034] The verification model execution unit 322 inputs the image (photographed image) acquired by the data acquisition unit 310 and executes the above-mentioned verification model in accordance with the settings made by the verification setting unit 320. The verification model outputs, for example, a frame indicating the area of ​​the detected object. However, this embodiment is not limited to a frame indicating the area of ​​the object, and other forms such as information indicating the shape of the area of ​​the object may be used. The verification model may also output information indicating the name of the detected object corresponding to the frame.

[0035] The erroneous detection determination unit 324 determines whether or not the detection result by the detection model contains an erroneous detection based on the detection result by the detection model and the detection result by the verification model. For example, when the difference between the detection result by the detection model and the detection result by the verification model is large enough to satisfy a certain condition, the erroneous detection determination unit 324 determines that the detection result by the detection model contains an erroneous detection.

[0036] Here, a case where it is determined that there is a false detection by the detection model will be described with reference to Fig. 4. The captured image 400 is a captured image acquired by the data acquisition unit 310. The captured image 400 is input to each of the detection model 402 and the verification model 404. The detection model 402 and the verification model 404 each detect an object in the captured image 400. In this example, the verification model 404 is a model that detects people more robustly than the detection model 402.

[0037] Detection result 406 shows a schematic example of an object detected by detection model 402. Frames 408 and 410 respectively show the regions of the objects detected by detection model 402. In detection result 406, frames 408 and 410 are superimposed on captured image 400 for ease of understanding. In this example, detection model 402 detects a person included in the region of frame 408 and a bicycle included in the region of frame 410. Detection region 420 shows a schematic view of only regions 422 and 424 of the object detected by detection model 402.

[0038] Meanwhile, the detection result 412 shows a schematic example of an object detected by the verification model 404. Frames 414 and 416 respectively show the areas of the objects detected by the verification model 404. In the detection result 412, the frames 414 and 416 are superimposed on the captured image 400 to facilitate understanding. In this example, the verification model 404 detects a person included in the area of ​​the frame 414 and a person included in the area of ​​the frame 416. The map image 426 is a binary image in which the numerical value "1" is assigned to areas 428 and 430 of the objects detected by the verification model 404, and the numerical value "0" is assigned to other areas.

[0039] The erroneous detection determination unit 324 extracts the difference between the map image 426 and the detection region 420. For example, a difference map image 432 is generated by assigning "1" to a region (non-overlapping region) where there is a difference between the object region of the map image 426 and the object regions 422 and 424. In the difference map image 432, a cluster of regions (difference region 434) occurs due to the difference between the object region detected by the detection model 402 and the object region detected by the verification model 404. For example, when the erroneous detection determination unit 324 determines that the difference region 434 is a size equal to or larger than a predetermined threshold, it determines that an erroneous detection exists in the detection result of the detection model 402. In this embodiment, a method of generating a difference map and determining the existence of an erroneous detection is used, but other methods such as a method of comparing only the number of object detections in the target image, a method of comparing the number of object detections in the target image with each coordinate position (for example, the barycentric coordinates of each object), and a method of comparing the number of object detections in the target image with each coordinate position and area can also be used.

[0040] Next, a case where it is determined that there is no erroneous detection by the detection model will be described with reference to Fig. 5. The captured image 500 is a captured image acquired by the data acquisition unit 310, and is the same as the captured image 400 shown in Fig. 4. The captured image 500 is input to each of the detection model 502 and the verification model 504. The detection model 502 and the verification model 504 each detect an object in the captured image 500. For example, the detection model 502 is a model that detects a person in an image, while the verification model 504 is a model that detects a person more robustly than the detection model 502.

[0041] Detection result 506 shows a schematic example of an object detected by detection model 502. Frames 508 and 510 respectively show the regions of the objects detected by detection model 502. In detection result 506, frames 508 and 510 are superimposed on captured image 500 for ease of understanding. In this example, detection model 502 detects a person included in the region of frame 508 and a person included in the region of frame 510. Detection region 520 shows a schematic state in which only regions 522 and 524 of the objects detected by detection model 502 are extracted.

[0042] Meanwhile, detection result 512 shows a schematic example of an object detected by verification model 504. Frames 514 and 516 respectively show the areas of the objects detected by verification model 504. In detection result 512, frames 514 and 516 are superimposed on captured image 500 for ease of understanding. In this example, verification model 504 detects a person included in the area of ​​frame 514 and a person included in the area of ​​frame 516. Map image 526 is a binary image in which regions 528 and 530 of objects detected by verification model 504 are assigned a numerical value "1" and other regions are assigned a numerical value "0."

[0043] The erroneous detection determination unit 324 extracts the difference between the map image 526 and the detection region 520. For example, a difference map image 532 is generated by assigning "1" to a region (non-overlapping region) where there is a difference between the object region of the map image 526 and the object regions 522 and 524. In the difference map image 532, there is no difference between the object region detected by the detection model 502 and the object region detected by the verification model 504. For this reason, the erroneous detection determination unit 324 determines that there is no erroneous detection in the detection result of the detection model 502, for example, because there is no difference region of a size equal to or larger than a predetermined threshold. In other words, it is determined that there is an erroneous detection when an object of a region of a size equal to or larger than a predetermined size is detected in the detection result of the verification model in a region where no object is detected in the detection result of the detection model.

[0044] The above-mentioned determination by the erroneous detection determination unit 324 can also be applied to the case where a plurality of verification models are used. For example, the erroneous detection determination unit 324 generates a map image by synthesizing (OR of the "1" region) map images 526 from a first verification model that detects people more robustly than the detection model and a second verification model that detects vehicles more robustly than the detection model. Then, a difference map image is generated between the synthesized map image and the object regions 422, 522, etc. That is, when an object of a region of a predetermined size or more is detected in at least one of the detection results of the first verification model and the detection results of the second verification model in a region where no object is detected in the detection result of the detection model, it can be determined that a erroneous detection exists. In this way, it becomes possible to determine erroneous detection related to a plurality of characteristics with a simple mechanism. That is, it becomes difficult to introduce complex implementation for each cause of erroneous detection, and the operation in the information processing device is simplified and the operation becomes easier to stabilize. In addition, as a method of using the first verification model and the second verification model in combination, in addition to the above, the second verification model may also be used based on the detection results of the first verification model.For example, when the detection reliability of each object, which is the detection result of the first verification model, is below or equal to a predetermined threshold, object detection is also performed using the second verification model, and the respective detection results are combined as described above.

[0045] 3 again. When the erroneous detection determination unit 324 determines that an erroneous detection has occurred, the notification unit 326 transmits a notification of the erroneous detection to an external device (e.g., the service provider terminal 106). The notification unit 326 may transmit a notification of the erroneous detection every time an erroneous detection occurs. Alternatively, the notification unit 326 may create notification information that records information related to the notification of the erroneous detection, and transmit the created notification information to the external device at predetermined time intervals.

[0046] The notification information may be, for example, a data structure 900 shown in Fig. 9. The data structure 900 may be configured by, for example, a frame number, details of the false detection, identification information of the detection model, identification information of the verification model, and identification information of the video.

[0047] The frame number is the number of a frame in the video where it is determined that a false detection exists. This frame is the frame number of the video identified by the "video identification information" of the data structure 900. The content of the false detection is, for example, information indicating what characteristic the false detection exists in. The notification unit 326 may describe the characteristic based on the detection result of the verification model when the false detection occurs. For example, the notification unit 326 may set the content of the false detection as "missed detection of a person" using the detection result of the verification model (the name of the object related to the frame 416, here a person) corresponding to the region where it is determined that a false detection exists (differential region 434). The identification information of the detection model and the identification information of the verification model are identification information of the models used when it is determined that a false detection exists.

[0048] By the service provider terminal 106 receiving the notification information generated by the notification unit 326, the service provider can easily understand what kind of erroneous detection has occurred in the detection model being operated and in what kind of image the erroneous detection has occurred. Therefore, by viewing the notification information, the service provider can easily improve the quality of the service using the detection model by identifying and supplementing the learning data used when retraining the detection model or by replacing the detection model.

[0049] The model data 330 is a database that stores data on the detection model and the validation model. The model data 330 includes information such as hyperparameters and learned parameters of each model.

[0050] The verification setting data 332 stores various information set by the verification setting unit 320 described above, such as identification information for the verification model used for verification, version information for the verification model, and information on the frequency of execution of the verification model.

[0051] Next, a series of operations for notifying a false detection in the information processing device 100 will be described with reference to FIG. 8. This process is realized by the processor 204 executing a computer program recorded in the storage 208. In the following description, a case is described in which the detection model and the verification model are executed at the same frequency (i.e., for all frames), but as described above, the verification model may be executed less frequently than the detection model. That is, as shown in FIG. 6, the detection model 600 may be executed for each frame (frame 610, etc.) in a video, while the verification model 602 may be executed less frequently (i.e., for frames 610, 614, 616).

[0052] In S801, the data acquisition unit 310 acquires an image (captured image) from the storage 208 or an external device. In S802, the detection model processing unit 312 executes object detection processing. For example, the detection model execution unit 314 executes a detection model on the acquired image to detect an object in the image.

[0053] Next, the verification model processing unit 318 executes an erroneous detection determination process. Specifically, in S803, the verification model execution unit 322 executes the verification model on the acquired image to detect an object in the image. Then, in S804, the erroneous detection determination unit 324 generates a map image (e.g., map image 426) showing the area of ​​the object detected by the verification model. In S805, the erroneous detection determination unit 324 excludes the area detected by the detection model from the area of ​​the object in the map image (i.e., performs the difference extraction described in FIG. 4 to generate the difference map image 432). In S806, the erroneous detection determination unit 324 calculates the area of ​​the cluster of the remaining area in the map image (difference area 434), and in S806, determines whether the area is equal to or greater than a threshold. If the erroneous detection determination unit 324 determines that the area is equal to or greater than the threshold, the process proceeds to S808, and if not, the process returns to S801.

[0054] In S808, the notification unit 326 generates, for example, notification information regarding the false detection, and notifies an external device of the presence of the false detection in real time or at a predetermined timing. In S809, the detection model processing unit 312 determines whether all images related to the video have been processed, and if it is determined that all images have been processed, ends this process, and if not, returns the process to S801.

[0055] The above-mentioned operation can also be applied to the case where a plurality of verification models are used. For example, as shown in FIG. 7, a first verification model 702 and a second verification model 704 among a plurality of verification models may be operated alternately. For example, the first verification model 702 is executed for frames 610 and 616, and the second verification model 704 is executed for frame 614. At this time, the combined execution frequency of the first verification model 702 and the second verification model 704 is lower than the execution frequency of the detection model 600.

[0056] As described above, in this embodiment, the detection model execution unit 314 uses a detection model that detects an object in an image to output a first detection result indicating an area in the image where an object is detected. Then, the verification model execution unit 322 uses a verification model whose characteristics regarding object detection are different from those of the detection model to output a second detection result indicating an area in the image where an object is detected. Furthermore, the erroneous detection determination unit 324 determines whether the difference between both detection results is large enough to satisfy a predetermined condition, and if the difference is large enough to satisfy the condition, determines that there is an erroneous detection in the detection of the object by the detection model. Furthermore, if there is an erroneous detection by the detection model, the notification unit 326 transmits notification information to, for example, the service provider terminal 106. In this way, in this embodiment, it is possible to discover an erroneous detection of a machine learning model using a general-purpose mechanism without requiring additional implementation according to the cause of the erroneous detection.

[0057] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0058] 100: information processing device, 102-1: surveillance camera, 102-2: user terminal, 104: communication device, 106: service provider terminal

Claims

1. An information processing device for detecting an object in an image, one or more processors; and a memory storing one or more instructions, the one or more instructions, when executed by the one or more processors, causing the information processing device to: using a first machine learning model to detect an object in an image, outputting a first detection result indicating an area in the image in which the object is detected; outputting a second detection result indicating an area within the image in which an object has been detected using a second machine learning model having a predetermined characteristic related to object detection different from that of the first machine learning model; and and when a difference between the first detection result and the second detection result satisfies a predetermined condition, notifying of the presence of a false positive in the detection of an object by the first machine learning model.

2. The information processing device according to claim 1 , wherein the second machine learning model has a predetermined property related to object detection that is more robust than the first machine learning model.

3. 3. The information processing device according to claim 2, wherein the notification includes determining that the predetermined condition is satisfied when an object of a predetermined size or larger is detected in the second detection result in an area where no object is detected in the first detection result.

4. The one or more instructions may further include: Acquire a series of frames of images; the first machine learning model outputs the first detection result for a first number of images per second among the plurality of frame images; The information processing device according to claim 1 , wherein the second machine learning model outputs the second detection result for a second number of images per second among the plurality of frame images, the second number being less than the first number.

5. The one or more instructions may further include: outputting a third detection result indicating an area within the image in which an object has been detected, using a third machine learning model having a second predetermined characteristic relating to object detection that is more robust than the first machine learning model; 4. The information processing device of claim 3, wherein the notification includes notifying of the presence of a false positive in the detection of an object by the first machine learning model when an object of an area equal to or larger than the predetermined size is detected in at least one of the second detection result and the third detection result in an area where no object is detected in the first detection result.

6. The information processing device according to claim 5 , wherein the notification includes notification of information regarding an object related to a false detection.

7. The information processing device according to claim 1 , wherein the second machine learning model has a property capable of detecting a specific type of object more robustly than the first machine learning model.

Citation Information

Patent Citations

  • Model training method and device, electronic equipment, storage medium and product

    CN114445682A

  • Attribute recognition system, learning server, and attribute recognition program

    JP2021009645A

  • Erroneous detection determination device, method, and program

    JP2022131761A

Cited By

  • Information processing device, information processing method, and recording medium

    WO2026014436A1