Object detection device, object detection method, and object detection program

Abstract

The goal is to reduce missed detections and false positives when using object detection models. [Solution] The detection control unit 21 inputs the input image to an object detection model 31 that detects objects with specified attributes from image data, causing the model to detect objects from the input image. When an object is detected by the object detection model 31, the determination control unit 22 inputs the input image to a determination model 32, which is a different trained model from the object detection model 31, and determines whether or not an object with the detection attribute, which is an attribute of the object detected by the object detection model 31, is included in the input image.

Description

Technical Field

[0001] The present disclosure relates to a technique for detecting an object of a specified attribute from image data.

Background Art

[0002] Patent Document 1 describes detecting an object included in image data using an object detection model learned using deep learning. In particular, Patent Document 1 describes detecting an object by a first method with a smaller amount of calculation than a method using deep learning, and then determining whether the detected object has a specified attribute by a second method using deep learning.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When detecting an object from image data, it is desirable to reduce the occurrence of false detection in which an object that should not be detected is detected and detection omission in which an object that should be detected is not detected. In the method described in Patent Document 1, false detection can be reduced to some extent by increasing the threshold value that serves as a criterion for determining that an object is specified by the second method, but the detection omission increases accordingly. Conversely, if the threshold value is lowered, detection omission can be reduced to some extent, but false detection increases accordingly. An object of the present disclosure is to be able to reduce detection omission and false detection when detecting an object using an object detection model.

Means for Solving the Problems

[0005] The object detection device according to the present disclosure is When an object detection model detects an object with specified attributes from image data, the object detection control unit inputs the input image to a determination model, which is a pre-trained model different from the object detection model, and determines whether or not an object with the detected attribute (an attribute of the object detected by the object detection model) is included in the input image. It is equipped with. [Effects of the Invention]

[0006] This disclosure makes it possible to reduce detection failures by setting a low threshold used when detecting objects in the object detection model. Furthermore, even with a low threshold, false positives can be kept to a minimum because a double check is performed using the object detection model and other models. [Brief explanation of the drawing]

[0007] [Figure 1] A diagram showing the configuration of the object detection device 10 according to Embodiment 1. [Figure 2] Flowchart of the processing of the object detection device 10 according to Embodiment 1. [Figure 3] An explanatory diagram of the input to the determination model 32 according to Embodiment 1. [Figure 4] An explanatory diagram of the output from the determination model 32 according to Embodiment 1. [Figure 5] Flowchart of the processing of the object detection device 10 according to Embodiment 2. [Figure 6] An explanatory diagram of the input to the determination model 32 according to Embodiment 2. [Figure 7] An explanatory diagram of the output from the determination model 32 according to Embodiment 2. [Figure 8] Flowchart of the processing of the object detection device 10 according to Embodiment 3. [Figure 9] An explanatory diagram of the input to the explanation generation model according to Embodiment 3. [Figure 10] An explanatory diagram of the output from the explanatory generation model according to Embodiment 3. [Figure 11]Explanatory diagram of the input to the attribute detection model according to Embodiment 3. [Figure 12] Explanatory diagram of the output from the attribute detection model according to Embodiment 3. [Figure 13] Flowchart of the processing of the object detection device 10 according to Embodiment 4. [Figure 14] Explanatory diagram of the input to the attribute detection model according to Embodiment 4. [Figure 15] Explanatory diagram of the output from the attribute detection model according to Embodiment 4. [Figure 16] Configuration diagram of the object detection device 10 according to Embodiment 5. [Figure 17] Flowchart of the processing of the object detection device 10 according to Embodiment 5. [Figure 18] Explanatory diagram of the input to the condition determination model 33 according to Embodiment 5. [Figure 19] Explanatory diagram of the output from the condition determination model 33 according to Embodiment 5.

Mode for Carrying Out the Invention

[0008] [[ID=3l]] Embodiment 1. ***Explanation of the Configuration*** Referring to FIG. 1, the configuration of the object detection device 10 according to Embodiment 1 will be described. The object detection device 10 is a computer. The object detection device 10 includes hardware such as a processor 11, a memory 12, a storage 13, and a communication interface Ⅳ. The processor 11 is connected to other hardware via signal lines and controls these other hardware.

[0009] The processor 11 is an IC that performs processing. IC is an abbreviation for Integrated Circuit. As specific examples, the processor 11 is a CPU, a DSP, or a GPU. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0010] Memory 12 is a storage device that temporarily stores data. As specific examples, Memory 12 is SRAM and DRAM. SRAM is the abbreviation of Static Random Access Memory. DRAM is the abbreviation of Dynamic Random Access Memory.

[0011] Storage 13 is a storage device that stores data. As a specific example, Storage 13 is an SSD. SSD is the abbreviation of Solid State Drive. Also, Storage 13 may be a portable recording medium such as a SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, flexible disk, optical disk, compact disk, Blu-ray (registered trademark) disk, or DVD. SD is the abbreviation of Secure Digital. DVD is the abbreviation of Digital Versatile Disk.

[0012] Communication interface 14 is an interface for communicating with an external device. As specific examples, Communication interface 14 is a port of Ethernet (registered trademark), USB, or HDMI (registered trademark). USB is the abbreviation of Universal Serial Bus. HDMI is the abbreviation of High-Definition Multimedia Interface.

[0013] The object detection device 10 includes, as functional components, a detection control unit 21, a determination control unit 22, and a notification unit 23. The functions of each functional component of the object detection device 10 are realized by software. The storage 13 stores a program for realizing the functions of each functional component of the object detection device 10. This program is read into the memory 12 by the processor 11 and executed by the processor 11. Thereby, the functions of each functional component of the object detection device 10 are realized.

[0014] Storage 13 stores the object detection model 31, which is a pre-trained model. The object detection model 31 is a model that detects objects with specified attributes from image data. Object detection model 31 is what is known as AI. AI stands for Artificial Intelligence. Object detection model 31 is a model constructed using deep learning.

[0015] Furthermore, the object detection device 10 is connected to the judgment model 32, which is a pre-trained model, via the communication interface 14. The classification model 32 is a so-called generative AI. The classification model 32 may be constructed using algorithms such as BERT and GPT. BERT stands for Bidirectional Encoder Representations from Transformers. GPT stands for Generative Pretrained Transformer. The learning model 112 may be constructed by combining multiple algorithms, including these algorithms.

[0016] Here, it is assumed that the object detection model 31 is stored in storage 13, and the judgment model 32 is connected via the communication interface 14. In other words, it is assumed that the object detection model 31 is inside the object detection device 10, and the judgment model 32 is outside the object detection device 10. However, both the object detection model 31 and the judgment model 32 may be inside the object detection device 10, or both the object detection model 31 and the judgment model 32 may be outside the object detection device 10.

[0017] In Figure 1, only one processor 11 was shown. However, there may be multiple processors 11, and multiple processors 11 may work together to execute programs that implement each function.

[0018] ***Explanation of operation*** Referring to Figures 2 to 4, the operation of the object detection device 10 according to Embodiment 1 will be explained. The operation procedure of the object detection device 10 according to Embodiment 1 corresponds to the object detection method according to Embodiment 1. Furthermore, the program that implements the operation of the object detection device 10 according to Embodiment 1 corresponds to the object detection program according to Embodiment 1.

[0019] Referring to Figure 2, the processing of the object detection device 10 according to Embodiment 1 will be explained. (Step S11: Image input processing) The detection control unit 21 sets the frame images that make up the video data as the input image.

[0020] (Step S12: Detection and control processing) The detection control unit 21 inputs an input image to the object detection model 31 and causes it to detect objects from the input image. At this time, the detection control unit 21 inputs one or more attributes to be detected along with the input image to the object detection model 31 and causes it to detect an object that has any of the one or more attributes. The detection control unit 21 acquires the detection results output from the object detection model 31. The detection results include a detection frame indicating the detected location of the detected object, attributes, and a confidence level indicating the certainty of the detection.

[0021] (Step S13: Confirmation process) The detection control unit 21 determines whether the confidence level included in the detection result obtained in step S12 is equal to or greater than the confirmation threshold. If the confidence level is above the confirmation threshold, the detection control unit 21 assumes that an object has been detected by the object detection model 31 and proceeds to step S14. On the other hand, if the confidence level is below the confirmation threshold, the detection control unit 21 terminates processing of the input image.

[0022] (Step S14: Decision control process) The determination control unit 22 inputs the input image to the determination model 32 and instructs it to determine whether or not an object with the detected attribute, which is an attribute of an object detected by the object detection model 31, is included in the input image. At this time, the determination control unit 22 instructs the determination model 32 to determine whether or not an object with the detected attribute is included in the input image and to create a description of the input image. For example, as shown in Figure 3, the judgment control unit 22 inputs the input image and attribute information indicating the detection attribute as prompts to the judgment model 32. At this time, the judgment control unit 22 inputs a prompt to the judgment model 32 instructing it to determine whether or not the object of the detection attribute is included in the input image and to create a description of the input image. In Figure 3, it is instructed to output in JSON Schema format. Outputting in JSON Schema format makes it easier to use the output data in subsequent processing. Here, ${attribute information} represents the attribute information specified in the input information (in the above example, "white cane"). In other words, "${attribute information}" is read as "white cane". Similarly, other ${***} are replaced with the information specified as *** in the input information. Furthermore, it is desirable that attribute information be clearly specified, for example, not simply "white cane," but rather "white cane for the visually impaired." If only "white cane" is specified, there is a possibility that white canes that are not for the visually impaired may also be judged as meeting the detection attribute. Therefore, if the attribute information is simply specified as "white cane," the judgment control unit 22 may use pre-configured information to replace "white cane" with "white cane for the visually impaired." Other examples of pre-configured information include replacing "wheelchair" with "wheelchair without an assistant" when the attribute information is specified, or replacing "safety vest" with "safety vest worn by construction workers" when the attribute information is specified. In this way, the judgment control unit 22 may use pre-configured information to replace the specified attribute information with more detailed attribute information.

[0023] The determination control unit 22 acquires the determination result and explanatory text output from the determination model 32. The determination result indicates whether or not the object with the detected attribute is included in the input image. When the input image and attribute information shown in Figure 3 are input as prompts, the judgment result and explanatory text are output in the format shown in Figure 4. In Figure 4, the judgment result is true, indicating that the object with the detected attribute is included in the input image, and the explanatory text is, "A woman is walking with a man while using a white cane..."

[0024] (Step S15: Detection and determination process) The determination control unit 22 determines whether the determination result obtained in step S14 indicates that an object with the detected attribute is included in the input image. If the determination result indicates that an object with the detection attribute is included in the input image, the determination control unit 22 proceeds to step S16. On the other hand, if the determination result does not indicate that an object with the detection attribute is included in the input image, the determination control unit 22 terminates processing of the input image.

[0025] (Step S16: Notification Processing) The notification unit 23 notifies that an object with the detection attribute has been detected and the explanatory text obtained in step 14. For example, the notification unit 23 displays on the display device used by the user of the object detection device 10 that an object with the detection attribute has been detected, along with the explanatory text.

[0026] ***Effects of Embodiment 1*** As described above, the object detection device 10 according to Embodiment 1, when an object is detected from the input image by the object detection model 31, uses the determination model 32 to determine whether or not an object with the detected attributes exists in the input image. This makes it possible to reduce missed detections and false positives. In other words, by setting a lower confirmation threshold used when detecting objects in the object detection model 31, it is possible to reduce missed detections. Furthermore, even when the confirmation threshold is set low, a double check is performed using the object detection model 31 and the judgment model 32, so it is possible to keep false positives to a minimum.

[0027] Furthermore, when the object detection device 10 according to Embodiment 1 notifies that an object with detection attributes has been detected, it also notifies the user of a description of the input image. By referring to the description, the user can more easily understand the situation, thereby improving usability.

[0028] The judgment model 32 is a so-called generative AI. Because processing with a generative AI takes time, it is difficult to perform in real time the process of determining whether or not an object is contained in all the frame images that make up the video. The object detection device 10 according to Embodiment 1 inputs the input image to the judgment model 32 only when an object is detected by the object detection model 31. Therefore, it becomes easier to realize detection processing using the judgment model 32, which is a generative AI, in real time.

[0029] Furthermore, the object detection device 10 according to Embodiment 1 can also be used for intrusion detection by devising a method for setting the attribute information to be input to the judgment model 32. For example, suppose we want to detect people who are on the track side of the yellow line on a train station platform. In this case, instead of simply setting the attribute information to "person," we can set it to "person on the track side of the yellow line," allowing the judgment model 32 to detect only people on the track side of the yellow line. Normally, when performing intrusion detection, it is necessary to set the area on the railway track side of the yellow line as the detection target area for each camera. However, with the object detection device 10 according to Embodiment 1, intrusion detection can be performed easily without such settings.

[0030] Embodiment 2. Embodiment 2 differs from Embodiment 1 in that, when the confidence level output from the object detection model 31 is above the confirmation threshold but not very high, the determination model 32 is made to determine which attributes of objects are included in the input image. Embodiment 2 explains this difference, and omits the explanation of the same points. In Embodiment 2, it is assumed that there are multiple attributes to be detected.

[0031] ***Explanation of operation*** Referring to Figure 5, the processing of the object detection device 10 according to Embodiment 2 will be explained. The processes from step S21 to step S23 are the same as the processes from step S11 to step S13 in Figure 2.

[0032] (Step S24: Selection determination process) The detection control unit 21 determines whether the confidence level included in the detection result obtained in step S22 is equal to or greater than the selection threshold. If the confidence level is equal to or greater than the selection threshold, the detection control unit 21 proceeds to step S25. In step S25, the processes from steps S14 to S16 in Figure 2 are executed. On the other hand, if the confidence level is less than the judgment threshold, the detection control unit 21 proceeds to step S26.

[0033] (Step S26: Decision control process) The determination control unit 22 inputs the input image to the determination model 32 and instructs it to determine whether or not an object with any of the multiple attributes to be detected is included in the input image. At this time, the determination control unit 22 instructs the determination model 32 to determine whether or not an object with any of the multiple attributes is included in the input image, and to create a description of the input image. For example, as shown in Figure 6, the determination control unit 22 inputs the input image and attribute information indicating multiple attributes of the object to be detected to the determination model 32. At this time, the determination control unit 22 determines whether or not an object with any of the multiple attributes is included in the input image, and also inputs a prompt to the determination model 32 instructing it to create a description of the input image.

[0034] The judgment control unit 22 acquires the judgment result and explanatory text output from the judgment model 32. The judgment result indicates the attributes of the object contained in the input image. When the input image and attribute information shown in Figure 6, along with the prompt, are input, the judgment result and explanatory text will be output in the format shown in Figure 7. In Figure 7, the judgment result indicates that a white cane is included in the input image, and the explanatory text reads, "A woman is walking with a man while using a white cane..."

[0035] (Step S27: Detection and determination process) The determination control unit 22 determines whether the determination result obtained in step S26 indicates that an object with one of the multiple attributes is included in the input image. If the determination result indicates that an object of any of the multiple attributes is included in the input image, the determination control unit 22 proceeds to step S28. On the other hand, if the determination result indicates that no object of any of the multiple attributes is included in the input image, the determination control unit 22 terminates processing of the input image.

[0036] (Step S28: Notification Processing) The notification unit 23 notifies that an object with the attributes indicated by the determination result obtained in step S26 has been detected, along with the explanatory text obtained in step 26.

[0037] ***Effects of Embodiment 2*** As described above, the object detection device 10 according to Embodiment 2 causes the determination model 32 to determine which attributes of the object are included in the input image when the confidence level is above the confirmation threshold but not very high. This makes it possible to appropriately identify the attributes of the object using the determination model 32 when there is a high probability that the object to be detected is included. Here, the judgment model 32 is a so-called generative AI. Although processing with a generative AI takes time, it can identify the attributes of an object with relatively high accuracy. The reason why the generation AI is highly accurate is likely because it can take surrounding information into account when identifying the attributes of the target to be detected. For example, if the attribute information is "white cane," even if a person is holding a white cane, if they are dressed for hiking, the AI ​​will not determine that a white cane has been detected.

[0038] ***Other configurations*** <Example 1> In Embodiment 2, an input image was input to the determination model 32, and it was made to determine whether or not an object with any of the multiple attributes to be detected was included in the input image. In this case, a situation where the confidence level is above the definitive threshold but not particularly high might occur when the detection result is split between two of the multiple attributes being detected. In other words, because it is difficult to distinguish between the two attributes, the confidence level may not be very high. Taking such cases into consideration, the determination control unit 22 may input the input image to the determination model 32 and have it determine whether or not an object with any of the attributes whose confidence level is above a lower limit is included in the input image. In other words, instead of having it determine whether or not any of the attributes to be detected is included, it may be determined whether or not an object with any of the attributes whose confidence level by the object detection model 31 was reasonably high is included. Alternatively, if there are multiple attributes whose difference in confidence level is less than a certain value, regardless of the level of confidence, the system can be configured to input those multiple attributes and determine whether any of them are included.

[0039] <Modification 2> If, in the past reference period or past reference time, the attributes of the object indicated by the judgment result obtained in step S26 are the same as the attributes indicated by the detection result obtained in step S22, the detection control unit 21 may lower the judgment threshold. Lowering the judgment threshold increases the number of cases where the process proceeds from step S24 to step S25 and decreases the number of cases where it proceeds to step S26, potentially reducing the occurrence of judgment errors by the generating AI in step S26 as a whole. Therefore, if there is a high probability that the obtained results will be the same, it is desirable to make step S25 more likely to be executed.

[0040] Embodiment 3. Embodiment 3 differs from Embodiment 1 in that it generates a descriptive text for the input image and determines from the descriptive text whether or not an object with the detection attribute is included in the input image. Embodiment 3 explains this difference, and omits explanations of the same points.

[0041] ***Explanation of operation*** Referring to Figure 8, the processing of the object detection device 10 according to Embodiment 3 will be explained. The processes from step S31 to step S33 are the same as the processes from step S11 to step S13 in Figure 2. The processes from step S36 and step S37 are the same as the processes from step S15 and step S16 in Figure 2.

[0042] Here, the judgment model 32 is described as including two separate generative AIs: an explanation generation model and an attribute detection model. The explanation generation model is a model specialized in image processing that takes image data as input and generates descriptive text for the image data. The attribute detection model is a model specialized in language processing that detects specified attributes from text. However, the judgment model 32 may be a standalone generative AI. In this case, you can substitute the explanation generation model, attribute detection model, and judgment model 32 in the following explanation.

[0043] (Step S34: Description generation process) The judgment control unit 22 receives the input image as input to the explanation generation model and causes it to create an explanatory text for the input image. For example, as shown in Figure 9, the determination control unit 22 inputs a prompt to the description generation model, along with the input image, instructing it to create a description of the input image. In this case, the length of the description may be specified to be longer so that it can express more detail than the notification description created in Embodiment 1.

[0044] The judgment control unit 22 retrieves the explanatory text output from the explanatory text generation model. When the input image and prompt shown in Figure 9 are input, an explanatory text like the one shown in Figure 10 is output.

[0045] (Step S35: Object detection process) The determination control unit 22 inputs the descriptive text acquired in step S34 and the detected attribute to the attribute detection model and causes it to determine from the descriptive text whether or not an object with the detected attribute is included in the input image. For example, as shown in Figure 11, the determination control unit 22 inputs a description and attribute information indicating the detected attribute to the attribute detection model. At this time, the determination control unit 22 determines whether or not the object of the detected attribute is included in the description and inputs a prompt to the attribute detection model instructing it to summarize the description.

[0046] The determination control unit 22 acquires the determination result output from the attribute detection model. The determination result indicates whether or not the detected attribute object is included in the description. When the description and attribute information shown in Figure 11 and the prompt are input, the determination result and the summarized description are output in the format shown in Figure 12.

[0047] ***Effects of Embodiment 3*** As described above, the object detection device 10 according to Embodiment 3 generates a descriptive text for the input image and determines from the descriptive text whether or not an object with detection attributes is included in the input image. Rather than making a determination directly from the input image, the explanation of the determination result is improved by creating a descriptive text before making the determination.

[0048] Furthermore, the object detection device 10 according to Embodiment 3 employs an explanation generation model specialized for image processing and an attribute detection model specialized for language processing. This makes it possible to increase processing speed and processing accuracy.

[0049] ***Other configurations*** <Variation 3> In Embodiment 3, a descriptive text was input to the attribute detection model, and it was made to determine from the text whether or not an object with the detected attribute was included in the description. However, the determination control unit 22 may determine from the description whether or not an object with the detected attribute was included in the input image by a simple word search. In other words, the determination control unit 22 may determine whether or not an object with the detected attribute was included in the input image by determining whether or not there is word in the description that indicates the detected attribute.

[0050] Embodiment 4. Embodiment 4 differs from Embodiment 3 in that, when the confidence level output from the object detection model 31 is above the confirmation threshold but not very high, the determination model 32 is made to determine which attributes of objects are included in the input image. Embodiment 4 explains this difference, and omits the explanation of the same points. In Embodiment 4, it is assumed that there are multiple attributes to be detected.

[0051] ***Explanation of operation*** Referring to Figure 13, the processing of the object detection device 10 according to Embodiment 4 will be explained. The processes from step S41 to step S43 are the same as the processes from step S31 to step S33 in Figure 8.

[0052] (Step S44: Selection determination process) Similar to step S24 in Figure 5, the detection control unit 21 determines whether the confidence level included in the detection result obtained in step S42 is equal to or greater than the selection threshold. If the confidence level is equal to or greater than the selection threshold, the detection control unit 21 proceeds to step S45. In step S45, the processes from steps S34 to S37 in Figure 8 are executed. On the other hand, if the confidence level is less than the judgment threshold, the detection control unit 21 proceeds to step S46.

[0053] (Step S46: Description generation process) Similar to step S34 in Figure 8, the decision control unit 22 inputs the input image to the explanation generation model and causes it to create an explanatory text for the input image. The decision control unit 22 then retrieves the explanatory text output by the explanation generation model.

[0054] (Step S47: Object detection process) The determination control unit 22 inputs the descriptive text obtained in step S34 to the attribute detection model and causes it to determine from the descriptive text whether or not an object with any of the multiple attributes to be detected is included in the input image. For example, as shown in Figure 14, a descriptive text and attribute information indicating multiple attributes of the object to be detected are input to the attribute detection model. At this time, the determination control unit 22 determines whether or not the object of the detected attribute is included in the input image, and also inputs a prompt to the attribute detection model instructing it to summarize the descriptive text.

[0055] The judgment control unit 22 acquires the judgment result output from the judgment model 32. The judgment result indicates the attributes of the object included in the input image. When the explanatory text and attribute information shown in Figure 14 and the prompt are input, the judgment result and summarized explanatory text are output in the format shown in Figure 15. In Figure 15, the judgment result indicates that a wheelchair is included in the input image.

[0056] (Step S48: Detection and determination process) Similar to step S27 in Figure 5, the determination control unit 22 determines whether the determination result obtained in step S47 indicates that an object with one of the multiple attributes is included in the input image. If the determination result indicates that an object of any of the multiple attributes is included in the input image, the determination control unit 22 proceeds to step S49. On the other hand, if the determination result indicates that no object of any of the multiple attributes is included in the input image, the determination control unit 22 terminates processing of the input image.

[0057] (Step S49: Notification processing) The notification unit 23 notifies that an object with the attributes indicated by the determination result obtained in step S47 has been detected, and also provides the summarized description obtained in step 47.

[0058] ***Effects of Embodiment 4*** As described above, the object detection device 10 according to Embodiment 4 causes the determination model 32 to determine which attributes of the object are included in the input image when the confidence level is above the confirmation threshold but not very high. This makes it possible to appropriately identify the attributes of the object using the determination model 32 when there is a high probability that the object to be detected is included.

[0059] Embodiment 5. Embodiment 5 differs from Embodiments 1 to 4 in that it determines whether or not to send a notification when an object is detected. Embodiment 5 explains this difference, while omitting explanations of the same points. Embodiment 5 describes a case where functionality is added to Embodiment 1. However, it is also possible to add functionality to Embodiments 2 to 4.

[0060] ***Explanation of the structure*** Referring to Figure 16, the configuration of the object detection device 10 according to Embodiment 5 will be described. The object detection device 10 is connected to the condition determination model 33 via a communication interface 14. The condition determination model 33, like the determination model 32, is a so-called generative AI. The condition determination model 33 is a model that determines conditions from text.

[0061] ***Explanation of operation*** Referring to Figure 17, the processing of the object detection device 10 according to Embodiment 5 will be explained. The processes from step S51 to step S55 are the same as the processes from step S11 to step S15 in Figure 2.

[0062] (Step S56: Conditional judgment process) The notification unit 23 inputs the explanatory text obtained in step S54 to the condition determination model 33, causing it to determine the notification level from the explanatory text. For example, as shown in Figure 18, a prompt is input to the conditional judgment model 33 along with a description, level classification settings, and notification enable / disable settings. In Figure 18, it is assumed that an object with attributes such as a wheelchair is identified, and two level classification settings are set: one where an assistant is present (no action required) and one where an assistant is not present (action required). The notification enable / disable setting is set to false (do not notify) if an assistant is present (no action required), and to true (notify) if an assistant is not present (action required).

[0063] The notification unit 23 acquires the judgment result output from the condition judgment model 33. The judgment result shows the level classification result and the notification feasibility result. When the information shown in Figure 18 is input, the level classification result and the notification feasibility result are shown in the format shown in Figure 19.

[0064] (Step S57: Notification determination process) The notification unit 23 determines whether the result of the determination obtained in step S56 indicates that notification should be made. If the notification unit 23 determines that notification is to be sent, it proceeds to step S58. On the other hand, if the notification unit 23 determines that notification is not to be sent, it terminates processing for the input image, returns to step S51, and moves on to processing for the next frame image.

[0065] (Step S58: Notification Processing) The notification unit 23 notifies that an object with the detection attribute has been detected, along with the explanatory text obtained in step 54.

[0066] ***Effects of Embodiment 5*** As described above, the object detection device 10 according to Embodiment 5 determines whether or not to send a notification when an object is detected. This reduces unnecessary notifications and alleviates the burden on the user who has to respond to notifications.

[0067] ***Other configurations*** <Modification 4> In Embodiment 5, the condition determination process in step S56 and the notification determination process in step S57 were executed after the processing by the determination control unit 22. Alternatively, the condition determination process in step S56 and the notification determination process in step S57 may be executed without processing by the determination control unit 22. In other words, if the confidence level in step S53 is equal to or greater than the confirmation threshold, the processing in steps S54 and S55 may be skipped and the process may proceed to step S56. In this case, the double-check using the judgment model 32 is not performed. However, a check using the conditional judgment model 33 is performed instead of the judgment model 32. Therefore, although their roles are slightly different, a certain level of double-checking effect is obtained.

[0068] <Modification 5> In the embodiments described above, each functional component was implemented in software. However, in Modification 5, each functional component may be implemented in hardware. The differences between this Modification 5 and the embodiments described above will be explained.

[0069] When each functional component is implemented in hardware, the object detection device 10 includes electronic circuits instead of the processor 11, memory 12, and storage 13. The electronic circuits are dedicated circuits that implement the functions of each functional component, as well as the functions of the memory 12 and storage 13.

[0070] Electronic circuits can include single circuits, complex circuits, programmed processors, parallel programmed processors, logic ICs, GAs, ASICs, and FPGAs. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. Each functional component may be implemented in a single electronic circuit, or it may be implemented by distributing each functional component across multiple electronic circuits.

[0071] <Variation 6> As a sixth variation, some of the functional components may be implemented in hardware, while others may be implemented in software.

[0072] The processor 11, memory 12, storage 13, and electronic circuitry are collectively referred to as the processing circuit. In other words, the function of each functional component is realized by the processing circuit.

[0073] Furthermore, the term "part" in the above explanation may be replaced with "circuit," "process," "procedure," "processing," or "processing circuit."

[0074] The various aspects of this disclosure are summarized below as an appendix. (Note 1) When an object detection model detects an object with specified attributes from image data, the object detection control unit inputs the input image to a determination model, which is a pre-trained model different from the object detection model, and determines whether or not an object with the detected attribute (an attribute of the object detected by the object detection model) is included in the input image. An object detection device equipped with the following features. (Note 2) The determination control unit inputs the input image to the determination model, generates a description of the input image, and determines from the description whether or not an object with a detection attribute, which is an attribute of an object detected by the object detection model, is included in the input image. The object detection device described in Appendix 1. (Note 3) The determination control unit receives the input image and the detection attribute as input to the determination model, generates a description of the input image, and determines from the description whether or not the object of the detection attribute is included in the input image. The object detection device described in Appendix 2. (Note 4) The object detection model outputs a detection result that includes the confidence level that an object with the specified attribute has been detected. If the confidence level included in the detection result is equal to or greater than the confirmation threshold, and the confidence level is equal to or greater than the selection threshold which is higher than the confirmation threshold, the determination control unit inputs the input image and the detection attributes to the determination model to generate a description of the input image and determines from the description whether or not an object of the detection attribute is included in the input image. If the confidence level is equal to or greater than the confirmation threshold, and the confidence level is less than the selection threshold, the determination control unit inputs the input image and a plurality of attributes to be detected to the determination model to generate a description of the input image and determines from the description whether or not an object of any of the plurality of attributes is included in the input image. The object detection device described in Appendix 3. (Note 5) The determination control unit inputs the descriptive text and the detected attribute to an attribute detection model that detects a specified attribute from a text, and causes the model to determine from the descriptive text whether or not an object with the detected attribute is included in the input image. The object detection device described in Appendix 2. (Note 6) The object detection model outputs a detection result that includes the confidence level that an object with the specified attribute has been detected. If the confidence level included in the detection result is equal to or greater than the confirmation threshold, and the confidence level is equal to or greater than the selection threshold which is higher than the confirmation threshold, the determination control unit inputs the description and the detection attribute to the attribute detection model to determine from the description whether or not an object of the detection attribute is included in the input image. If the confidence level is equal to or greater than the confirmation threshold, and the confidence level is less than the selection threshold, the determination control unit inputs the description and a plurality of attributes to be detected to the attribute detection model to determine from the description whether or not an object of any of the plurality of attributes is included in the input image. The object detection device described in Appendix 5. (Note 7) The object detection device further, When the determination control unit determines that an object with the detection attribute is included in the input image, the notification unit notifies that an object with the detection attribute has been detected, along with the explanatory text. An object detection device as described in any one of the appendices 2 to 5, comprising: (Note 8) The object detection device further, When the determination control unit determines that an object with any of the aforementioned multiple attributes is included in the input image, the notification unit notifies that an object with any of the aforementioned multiple attributes has been detected, along with the explanatory text. An object detection device as described in Appendix 4 or 6, comprising the features described herein. (Note 9) The object detection device further, A notification unit inputs the explanatory text into a conditional judgment model that determines conditions from text, determines the notification level from the explanatory text, and sends a notification in a manner corresponding to the determined notification level. An object detection device according to any one of the appendices 2 to 8, comprising: (Note 10) An object detection method in which, when a computer detects an object from an input image using an object detection model that detects objects with specified attributes from image data, the computer inputs the input image to a determination model, which is a pre-trained model different from the object detection model, and determines whether or not an object with a detection attribute, which is an attribute of the object detected by the object detection model, is included in the input image. (Note 11) When an object is detected in an input image by an object detection model that detects objects with specified attributes from image data, a determination control process is performed to input the input image to a determination model, which is a pre-trained model different from the object detection model, and to determine whether or not an object with the detected attribute, which is an attribute of the object detected by the object detection model, is included in the input image. An object detection program that makes a computer function as an object detection device.

[0075] The embodiments and variations of this disclosure have been described above. Some of these embodiments and variations may be implemented in combination. Alternatively, some or all of them may be implemented in part. However, this disclosure is not limited to the embodiments and variations described above, and various modifications are possible as needed. [Explanation of symbols]

[0076] 10 Object detection device, 11 Processor, 12 Memory, 13 Storage, 14 Communication interface, 21 Detection control unit, 22 Judgment control unit, 23 Notification unit, 31 Object detection model, 32 Judgment model, 33 Condition judgment model.

Claims

1. When an object detection model detects an object with specified attributes from image data, the object detection control unit inputs the input image to a determination model, which is a pre-trained model different from the object detection model, and determines whether or not an object with the detected attribute (an attribute of the object detected by the object detection model) is included in the input image. An object detection device equipped with the following features.

2. The determination control unit inputs the input image to the determination model, generates a description of the input image, and determines from the description whether or not an object with a detection attribute, which is an attribute of an object detected by the object detection model, is included in the input image. The object detection device according to claim 1.

3. The determination control unit receives the input image and the detection attribute as input to the determination model, generates a description of the input image, and determines from the description whether or not the object of the detection attribute is included in the input image. The object detection device according to claim 2.

4. The object detection model outputs a detection result that includes the confidence level that an object with the specified attribute has been detected. If the confidence level included in the detection result is equal to or greater than the confirmation threshold, and the confidence level is equal to or greater than the selection threshold which is higher than the confirmation threshold, the determination control unit inputs the input image and the detection attributes to the determination model to generate a description of the input image and determines from the description whether or not an object of the detection attribute is included in the input image. If the confidence level is equal to or greater than the confirmation threshold, and the confidence level is less than the selection threshold, the determination control unit inputs the input image and a plurality of attributes to be detected to the determination model to generate a description of the input image and determines from the description whether or not an object of any of the plurality of attributes is included in the input image. The object detection device according to claim 3.

5. The determination control unit inputs the descriptive text and the detected attribute to an attribute detection model that detects a specified attribute from a text, and causes the model to determine from the descriptive text whether or not an object with the detected attribute is included in the input image. The object detection device according to claim 2.

6. The object detection model outputs a detection result that includes the confidence level that an object with the specified attribute has been detected. If the confidence level included in the detection result is equal to or greater than the confirmation threshold, and the confidence level is equal to or greater than the selection threshold which is higher than the confirmation threshold, the determination control unit inputs the description and the detection attribute to the attribute detection model to determine from the description whether or not an object of the detection attribute is included in the input image. If the confidence level is equal to or greater than the confirmation threshold, and the confidence level is less than the selection threshold, the determination control unit inputs the description and a plurality of attributes to be detected to the attribute detection model to determine from the description whether or not an object of any of the plurality of attributes is included in the input image. The object detection device according to claim 5.

7. The object detection device further, When the determination control unit determines that an object with the detection attribute is included in the input image, the notification unit notifies that an object with the detection attribute has been detected, along with the explanatory text. The object detection device according to claim 2, comprising:

8. The object detection device further, When the determination control unit determines that an object with any of the aforementioned multiple attributes is included in the input image, the notification unit notifies that an object with any of the aforementioned multiple attributes has been detected, along with the explanatory text. The object detection device according to claim 4, comprising:

9. The object detection device further, A notification unit inputs the explanatory text into a conditional judgment model that determines conditions from text, determines the notification level from the explanatory text, and sends a notification in a manner corresponding to the determined notification level. The object detection device according to claim 2, comprising:

10. An object detection method in which, when a computer detects an object from an input image using an object detection model that detects objects with specified attributes from image data, the computer inputs the input image to a determination model, which is a pre-trained model different from the object detection model, and determines whether or not an object with a detection attribute, which is an attribute of the object detected by the object detection model, is included in the input image.

11. When an object is detected in an input image by an object detection model that detects objects with specified attributes from image data, the input image is input to a determination model, which is a pre-trained model different from the object detection model, to determine whether or not an object with the detected attribute (an attribute of the object detected by the object detection model) is included in the input image. An object detection program that makes a computer function as an object detection device.

Images