Computer, object detection method, and system
The system addresses the inefficiencies of conventional object detection by using a detection model to track and verify objects based on work-specific rules, enhancing the detection of abandoned items in various operational contexts.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- HITACHI LTD
- Filing Date
- 2023-03-22
- Publication Date
- 2026-05-13
AI Technical Summary
Conventional object detection systems fail to account for the specific nature of the work being performed, leading to inefficiencies in identifying abandoned objects in different operational contexts.
A system utilizing a detection model to identify objects in images, track their presence over time, and verify them based on business-specific rules, incorporating object tracking and event detection to manage background objects and detect abandoned items effectively.
Enables efficient detection of abandoned objects relevant to the work environment by reducing processing load and improving accuracy through business-specific verification processes.
Smart Images
Figure 0007857888000001 
Figure 0007857888000002 
Figure 0007857888000003
Abstract
Description
Technical Field
[0001] The present invention relates to a technique for detecting an object depicted in an image.
Background Art
[0002] In a workplace where operations such as luggage transportation are carried out, there are cases where leaving luggage unattended becomes a problem. As a technique for monitoring an unattended object, the technique described in Patent Document 1 is known. Patent Document 1 discloses an apparatus that “the object tracking means 41 detects an object image from a monitoring image, identifies the object images detected at successive times, stores an object image history 31 in which the object images are historized for each identified object in the storage unit 3, the suspicious object detection means 42 analyzes the object image history 31 to detect a suspicious object that has been stationary for a period exceeding the stationary determination time, the abandonment owner detection means 43 refers to the object image history 31 within a predetermined period until the object image of the suspicious object starts to be detected, and detects all objects detected in the detection area of the suspicious object as candidate objects for the abandonment owner, and the case information output means 44 selectively outputs the information of the object detected by the abandonment owner detection means 43 from the object image history 31.”
[0003] In Patent Document 1, the type of the object has not been identified. On the other hand, a technique for detecting the type and position of an object depicted in an image using a model such as a CNN (Convolutional Neural Network) is known.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0005] By using Patent Document 1 and the model, it is possible to detect abandoned objects of a specific type. The purposes of on-site monitoring vary, and the conditions for detecting objects differ depending on the type of work performed on-site. Conventional technology does not detect objects that reflect the type of work performed.
[0006] The present invention aims to provide a technology for detecting abandoned objects, taking into account the nature of the work. [Means for solving the problem]
[0007] A representative example of the invention disclosed in this application is as follows: It has a calculation unit and a storage device connected to the calculation unit. It is a computer, The aforementioned storage device is A detection model that detects objects contained in an image memory death, The aforementioned computing device is Images of the space where the work is performed are acquired, and objects contained in the images are detected using the aforementioned detection model. The system detects the occurrence of an event representing a change in the environment of the space, detects objects included in the image acquired at the time of the change in the environment of the space, registers the detected objects in the background object list, and after the occurrence of the event, An object tracking process is performed to track an object using the object detection results of multiple images input in chronological order, and the object detected from the latest image is tracked. From among them, identify objects that are not registered in the background object list, and for the identified objects, The system calculates the detection duration, which represents the time the object has been continuously detected. Objects with a detection duration greater than a threshold are identified as objects to be verified. A verification process is then performed to determine whether or not the object to be verified is a detection target object in the business, based on a judgment rule that takes into account the content of the business. [Effects of the Invention]
[0008] According to the present invention, a technology for detecting abandoned objects that takes into account the nature of the work can be realized. Other problems, configurations, and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]
[0009] [Figure 1] This diagram shows an overview of the monitoring system in Example 1. [Figure 2] This figure shows an example of the hardware configuration of a computer that implements the apparatus of Example 1. [Figure 3] It is a diagram showing an example of the software configuration of the verification apparatus of Example 1. [Figure 4] It is a diagram showing an example of the software configuration of the recording apparatus of Example 1. [Figure 5] It is a diagram showing an example of the software configuration of the visualization apparatus of Example 1. [Figure 6] It is a diagram showing an example of the background object information of Example 1. [Figure 7A] It is a diagram showing an example of the processing result information of Example 1. [Figure 7B] It is a diagram showing an example of the processing result information of Example 1. [Figure 8A] It is a flowchart explaining an example of the processing executed by the object detection application of Example 1. [Figure 8B] It is a flowchart explaining an example of the processing executed by the object detection application of Example 1. [Figure 9] It is a diagram explaining an example of the method for registering a background object in Example 1. [Figure 10] It is a flowchart explaining an example of the verification processing executed by the object detection application of Example 1. [Figure 11] It is a diagram showing an example of the screen presented by the visualization apparatus of Example 1. [Figure 12] It is a flowchart explaining an example of the verification processing executed by the object detection application of Example 2. [Figure 13] It is a diagram explaining an example of object detection by the verification processing of Example 2. [Figure 14] It is a flowchart explaining an example of the verification processing executed by the object detection application of Example 3. [Figure 15] It is a diagram explaining an example of object detection by the verification processing of Example 3. [Figure 16] It is a diagram showing an example of the configuration of the system of Example 4. [Figure 17] It is a diagram showing an example of the functional configuration of the detection model learning apparatus of Example 4. [Figure 18]It is a flowchart for explaining an example of the generation process of the detection model executed by the detection model learning device of Example 4.
Mode for Carrying Out the Invention
[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not to be construed as being limited to the description of the embodiments shown below. It will be readily understood by those skilled in the art that the specific configuration can be changed without departing from the spirit or gist of the present invention.
[0011] In the configuration of the invention described below, the same or similar configurations or functions are denoted by the same reference numerals, and redundant descriptions are omitted.
[0012] The notations such as "first", "second", "third", etc. in this specification and the like are attached to identify the components, and do not necessarily limit the number or order.
[0013] The positions, sizes, shapes, and ranges of each configuration shown in the drawings and the like may not represent the actual positions, sizes, shapes, and ranges in order to facilitate the understanding of the invention. Therefore, in the present invention, it is not limited to the positions, sizes, shapes, and ranges disclosed in the drawings and the like.
Embodiment
[0014] FIG. 1 is a diagram showing an outline of the monitoring system of Example 1. FIG. 2 is a diagram showing an example of the hardware configuration of a computer that realizes the device of Example 1.
[0015] The monitoring system 10 is a system that monitors a monitoring area 11 such as a logistics base. A camera 104 for photographing the monitoring area 11 is installed in the monitoring area 11, and a detection target area 12 is set. The monitoring system 10 is connected to the camera 104 via a network 105 such as a LAN (Local Area Network).
[0016] The monitoring system 10 includes a verification device 100, a recording device 101, a visualization device 102, and an input / output device 103.
[0017] The recording device 101 records images. The verification device 100 detects and tracks objects. The visualization device 102 presents the processing results of the verification device 100 and the images recorded by the recording device 101 to the user. The input / output device 103 includes a keyboard, mouse, touch panel, and display.
[0018] The verification device 100, the recording device 101, and the visualization device 102 are implemented using a computer 200 with a hardware configuration as shown in Figure 2, for example. The computer 200 includes a CPU 201, memory 202, a network interface 203, and an I / O interface 204. The computer 200 may also include storage devices such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).
[0019] Memory 202 stores programs and information. Memory 202 is also used as a work area.
[0020] CPU201 is an example of an arithmetic unit that executes programs stored in memory202. By executing processes according to the program, CPU201 operates as a functional unit (module) that realizes a specific function. In the following explanation, when a functional unit is the subject of a description of its processing, it indicates that CPU201 is executing the program that realizes that functional unit.
[0021] The network interface 203 communicates with external devices via the network. The I / O interface 204 connects to input / output devices.
[0022] In Example 1, the objective is to detect packages left in the detection target area 12 in order to monitor for forgotten packages and packages left unattended for extended periods during the loading process.
[0023] Figure 3 shows an example of the software configuration of the verification device 100 in Example 1.
[0024] The verification device 100 has an object detection application 300 and also stores detection model information 301, background object information 302, and processing result information 303.
[0025] Detection model information 301 is information for managing the detection model that detects objects contained in the image. The detection model is, for example, a CNN, and outputs the type of object, an image of the rectangular area surrounding the object, and the coordinates of the rectangle. Note that the present invention is not limited to the type of detection model. Background object information 302 is information for managing objects (background objects) contained in the image treated as the background of the monitoring area 11. Processing result information 303 is information for managing the processing results of the object detection application 300.
[0026] The object detection application 300 detects and verifies abandoned objects in the monitoring area 11. The object detection application 300 includes an image acquisition unit 310, an object detection unit 311, an event detection unit 312, a background object registration unit 313, an object tracking unit 314, an abandoned object detection unit 315, a verification unit 316, and an output unit 317.
[0027] The image acquisition unit 310 acquires an image from the camera 104. The image acquisition unit 310 may perform image transformations as necessary. The object detection unit 311 detects objects included in the image using a detection model. The event detection unit 312 detects the occurrence of an event that triggers the detection of abandoned objects. The background object registration unit 313 registers background objects. The object tracking unit 314 performs object tracking processing. Object tracking processing is the process of estimating the movement of objects included in the image. The abandoned object detection unit 315 detects abandoned objects. The verification unit 316 verifies whether the abandoned object is an object to be detected in the business. The output unit 317 outputs the processing results.
[0028] Furthermore, the functional units included in the object detection application 300 may be combined into a single functional unit, or a single functional unit may be divided into multiple functional units.
[0029] Figure 4 shows an example of the software configuration of the recording device 101 in Embodiment 1.
[0030] The recording device 101 has a recording application 400 and also holds an image database 401.
[0031] Image DB401 is a database for managing images acquired from camera 104.
[0032] The recording application 400 saves and plays back images. The recording application 400 includes an image recording unit 410, an image playback unit 411, and an image output unit 412.
[0033] The image recording unit 410 acquires images from the camera 104 and records them in the image database 401. The image playback unit 411 acquires multiple images from the image database 401 and generates a video. The image output unit 412 outputs either an image or a video.
[0034] Furthermore, the functional units included in the recording application 400 may either combine multiple functional units into a single functional unit, or divide a single functional unit into multiple functional units.
[0035] Figure 5 shows an example of the software configuration of the visualization device 102 in Example 1.
[0036] The visualization device 102 has a visualization application 500 and also holds processing result information 501.
[0037] The visualization application 500 displays the processing results of the verification device 100 and the images managed by the recording device 101. The visualization application 500 includes an image retrieval unit 510, an image generation unit 511, and a processing result recording unit 512.
[0038] The image retrieval unit 510 acquires images or videos from the recording device 101. The image generation unit 511 generates images to be presented to the user using the images or videos acquired from the recording device 101. The image generation unit 511 also generates display information to present various information to the user. The processing result recording unit 512 acquires processing results from the verification device 100 and presents them to the user. The processing result recording unit 512 also accepts user modifications to the processing results.
[0039] Furthermore, the functional units included in the visualization application 500 may be combined into a single functional unit, or a single functional unit may be divided into multiple functional units.
[0040] Figure 6 shows an example of background object information 302 from Example 1.
[0041] The background object information 302 stores, for example, a table 600. Table 600 stores entries that include an ID 601, a position 602, and an object image 603. There is one entry for each background object. Note that the fields included in an entry are examples only and are not limited to these.
[0042] ID601 is a field that stores the ID of the background object. Position602 is a field that stores the position of the background object included in the image. Object image603 is a field that stores a partial image containing the background object, which is extracted from the image.
[0043] Figures 7A and 7B show an example of the processing result information 303 from Example 1.
[0044] The processing result information 303 stores the detection target object table 700 and the false detection object table 710. The detection target object table 700 stores the detection results for the detection target objects. The false detection object table 710 stores the detection results for objects that were mistakenly detected as detection target objects (false detection objects).
[0045] The detection target object table 700 stores entries that include ID 701, timestamp 702, object image 703, and registration type 704. There is one entry for each detected object. Note that the fields included in an entry are examples only and are not limited to these.
[0046] ID701 is a field that stores the ID of the entry. Timestamp702 is a field that stores the date and time the image in which the object was detected was acquired. Object image703 is a field that stores a partial image containing the object that was cut out from the image. Registration type704 is a field that stores information indicating the registration entity of the detected object table700. If the entry is registered by the object detection application300, "Automatic" is stored in registration type704, and if the entry is registered as a result of a user modification operation, "Manual" is stored in registration type704.
[0047] The false detection object table 710 stores entries that include ID 711, timestamp 712, object image 713, and registration type 714. ID 711, timestamp 712, object image 713, and registration type 714 are the same fields as ID 701, timestamp 702, object image 703, and registration type 704.
[0048] Figures 8A and 8B are flowcharts illustrating an example of the process performed by the object detection application 300 in Example 1. Figure 9 is a diagram illustrating an example of the method for registering background objects in Example 1.
[0049] When the object detection application 300 acquires an image from the camera 104, it executes the process described below. Furthermore, when it acquires a video from the camera 104, the object detection application 300 executes the process described below for each image included in the video.
[0050] Furthermore, the verification device 100 in Example 1 is configured to retain information about the detection target area 12 in the image.
[0051] The object detection application 300 uses a detection model to perform object detection processing to detect objects from an image (step S101).
[0052] The object detection application 300 determines whether or not an event has occurred based on the detection result (step S102). In Example 1, the opening of the door of a storage room where goods are stored is considered the occurrence of an event. Note that the method for detecting the occurrence and termination of an event is just one example and is not limited thereto. For example, the occurrence and termination of an event may be detected based on time.
[0053] One possible method for determining the occurrence of an event is to use a model that has learned image features before and after the event occurred. This model may be a detection model or a model different from a detection model.
[0054] If no event has occurred, the object detection application 300 determines whether the event flag, which indicates the occurrence of an event, is ON or OFF (step S103). The event flag is initially set to OFF.
[0055] If the event flag is OFF, the object detection application 300 terminates processing.
[0056] If the event flag is ON, the object detection application 300 changes the event flag to OFF, initializes the background object information 302 (step S104), and then terminates the process.
[0057] If an event occurs, the object detection application 300 determines whether the event flag is ON or OFF (step S105).
[0058] If the event flag is ON, the object detection application 300 proceeds to step S107.
[0059] If the event flag is OFF, the object detection application 300 updates the background object information 302 (step S106). At this time, the object detection application 300 changes the event flag to ON. After that, the object detection application 300 proceeds to step S107.
[0060] Specifically, the object detection application 300 generates background object information 302 based on the processing results of the object detection process.
[0061] Here, we will explain the generation of background object information 302 using Figure 9. In the image at t=k-1, the door is closed, so background object information 302 is not generated. In the image at t=k, the door is open, so background object information 302 is generated. The object detection application 300 performs the processing from step S105 onwards for images from t=k onwards.
[0062] By registering objects included in the image at the time of the event as background objects, it is possible to efficiently detect newly appearing objects in the image.
[0063] In step S107, the object detection application 300 performs object tracking (step S107). By performing object tracking, an association is made between the objects in the currently processed image and the objects in past images. This makes it possible to track the objects in the image. Although a detailed explanation of object tracking is omitted as it is a well-known technique, for example, methods using Kalman filters and image features can be considered.
[0064] The object detection application 300 selects one object from among the detected objects (step S108).
[0065] The object detection application 300 determines whether the selected object is a background object or not based on the background object information 302 and the results of the object tracking process (step S109).
[0066] If the selected object is a background object, the object detection application 300 proceeds to step S116.
[0067] If the selected object is not a background object, the object detection application 300 calculates the detection duration based on the results of the object tracking process (step S110).
[0068] Specifically, the object detection application 300 calculates the detection duration as the difference between the timestamp of the image in which the selected object was first detected and the timestamp of the current image.
[0069] The object detection application 300 determines whether the detection duration is greater than a threshold (step S111). In other words, it determines whether the object has been left unattended for a certain period of time.
[0070] If the detection duration is below the threshold, the object detection application 300 proceeds to step S116.
[0071] If the detection duration is greater than the threshold, the object detection application 300 performs a verification process (step S112). Details of the verification process will be described later.
[0072] The object detection application 300 determines whether the selected object is a target object based on the results of the verification process (step S113).
[0073] If the selected object is a target object, the object detection application 300 registers an entry in the target object table 700 (step S114), and then proceeds to step S116.
[0074] If the selected object is not a target object, the object detection application 300 registers an entry in the false detection object table 710 (step S115), and then proceeds to step S116.
[0075] In step S116, the object detection application 300 determines whether processing has been completed for all detected objects (step S116).
[0076] If processing is not complete for all detected objects, the object detection application 300 returns to step S108. If processing is complete for all detected objects, the object detection application 300 terminates processing.
[0077] Figure 10 is a flowchart illustrating an example of the verification process performed by the object detection application 300 in Example 1.
[0078] The object detection application 300 acquires information about the detection target area 12 (step S201).
[0079] The object detection application 300 determines whether the selected object is present in the detection area 12 (step S202). This determination is made based on the object's current position.
[0080] If the selected object is present in the detection area 12, the object detection application 300 determines that the selected object is a target object (step S203), and then terminates the verification process.
[0081] If the selected object is not present in the detection area 12, the object detection application 300 determines that the selected object is a false positive (step S204), and then terminates the verification process.
[0082] Figure 11 shows an example of the screen displayed by the visualization device 102 of Example 1.
[0083] Screen 1100 is the screen that the visualization device 102 displays to the input / output device 103. Screen 1100 includes a table of detected objects 1101 and a table of falsely detected objects 1102.
[0084] The detected object table 1101 stores entries including ID 1111, timestamp 1112, object image 1113, registration type 1114, playback 1115, and correction 1116. ID 1111, timestamp 1112, object image 1113, and registration type 1114 are the same fields as ID 701, timestamp 702, object image 703, and registration type 704. Playback 1115 is a field that displays operation buttons for playing a video containing the image in which the detected object was detected. Correction 1116 is a field that displays operation buttons for the user to correct a detected object to a false positive.
[0085] The false detection object table 1102 stores entries including ID 1121, timestamp 1122, object image 1123, registration type 1124, playback 1125, and correction 1126. ID 1121, timestamp 1122, object image 1123, and registration type 1124 are the same fields as ID 711, timestamp 712, object image 713, and registration type 714. Playback 1125 is a field that displays operation buttons for playing a video containing the image in which the false detection object was detected. Correction 1116 is a field that displays operation buttons for the user to correct the false detection object to the target object.
[0086] According to Example 1, the verification device 100 can detect objects that take into account the nature of the work. By registering objects included in the image at the time of the event as background objects, the object to be verified can be efficiently identified. By performing object tracking in response to the occurrence of an event, the processing load on the verification device 100 can be reduced while enabling monitoring of objects that are relevant to the work. [Examples]
[0087] In Example 2, the verification process differs from that of Example 1. The following description focuses on the differences between Example 2 and Example 1.
[0088] In Example 2, the objective is to detect cargo that has been removed from the storage area and left unattended, in order to monitor for forgotten cargo loading and cargo being left unattended for extended periods during cargo loading operations.
[0089] The configuration of the monitoring system 10 in Example 2 is the same as that of Example 1. The functional configurations of the verification device 100, recording device 101, and visualization device 102 in Example 2 are the same as those in Example 1.
[0090] Figure 12 is a flowchart illustrating an example of the verification process performed by the object detection application 300 in Example 2. Figure 13 is a diagram illustrating an example of object detection using the verification process in Example 2.
[0091] The object detection application 300 acquires location information of the door (reference point) of the storage compartment where the cargo is stored (step S301). The door's location information is pre-set.
[0092] The object detection application 300 identifies the location where the selected object first appeared (appearance location) based on the tracking results of the selected object (step S302).
[0093] The object detection application 300 calculates the distance between the door's position and its appearance position (step S303).
[0094] The object detection application 300 determines whether the calculated distance is less than a threshold (step S304).
[0095] If the calculated distance is less than the threshold, the object detection application 300 determines that the selected object is the object to be detected (step S305), and then terminates the verification process.
[0096] If the calculated distance is greater than or equal to a threshold, the object detection application 300 determines that the selected object is a falsely detected object (step S306), and then terminates the verification process.
[0097] In Example 2, as shown in Figure 13, the objective is to detect cargo that has been removed from the hangar and left unattended as the target object. The object detection application 300 uses the appearance location of the object to determine whether or not it is cargo that has been removed from the hangar. Trajectory 1310 represents the tracking result of cargo 1300, and trajectory 1311 represents the tracking result of cargo 1301. Since the distance between the appearance location of cargo 1300 (starting point of trajectory 1310) and the door is sufficiently close, cargo 1300 is determined to be cargo that has been removed from the hangar. Since the distance between the appearance location of cargo 1301 (starting point of trajectory 1311) and the door is far, cargo 1301 is determined not to be cargo that has been removed from the hangar. [Examples]
[0098] In Example 3, the verification process differs from that of Example 1. The following description focuses on the differences between Example 3 and Example 1.
[0099] In Example 3, the objective is to detect unattended, low-temperature cargo in order to monitor for forgotten cargo loading and cargo being left unattended for extended periods during cargo loading operations.
[0100] The configuration of the monitoring system 10 in Example 3 is the same as in Example 1. However, in Example 3, a thermographic camera 1500 is installed in the monitoring area 11. The functional configurations of the verification device 100, recording device 101, and visualization device 102 in Example 3 are the same as in Example 1.
[0101] Figure 14 is a flowchart illustrating an example of the verification process performed by the object detection application 300 in Example 3. Figure 15 is a diagram illustrating an example of object detection using the verification process in Example 3.
[0102] The object detection application 300 obtains the temperature of the selected object based on the image acquired from the thermal imaging camera 1500 (step S401).
[0103] The object detection application 300 determines whether the object's temperature is below a threshold (step S402).
[0104] If the object's temperature is below the threshold, the object detection application 300 determines that the selected object is the object to be detected (step S403), and then terminates the verification process.
[0105] If the object's temperature is above a threshold, the object detection application 300 determines that the selected object is a false positive (step S404), and then terminates the verification process.
[0106] In Example 3, as shown in Figure 15, the objective is to detect packages that have been removed from a low-temperature storage facility and left unattended as target objects. The object detection application 300 uses the temperature of the object to determine whether or not it is a low-temperature package. [Examples]
[0107] Example 4 describes a method for training a detection model using the processing results of the verification device 100.
[0108] Figure 16 shows an example of the system configuration of Example 4.
[0109] The system consists of multiple monitoring systems 10 and a detection model learning device 1600. One monitoring system 10 exists for each district, and the detection model learning device 1600 connects to the multiple monitoring systems 10 via a network 1601 such as a LAN (Local Area Network).
[0110] The detection model learning device 1600 acquires false detection object data from each monitoring system 10. The false detection object data corresponds to one entry in the false detection object table 710. The detection model learning device 1600 performs the detection model learning process using the existing training data and the acquired false detection object data. The detection model learning device 1600 distributes the new detection model to each monitoring system 10.
[0111] The verification process of the monitoring system 10 may be any of the examples in Examples 1 to 3.
[0112] The detection model learning device 1600 may also acquire detection target object data from each monitoring system 10. The detection target object data corresponds to one entry in the detection target object table 700.
[0113] Figure 17 shows an example of the functional configuration of the detection model learning device 1600 of Example 4.
[0114] The detection model learning device 1600 has a learning application 1700 and also holds detection model information 1701 and a learning data DB 1702.
[0115] Detection model information 1701 is information for managing the detection model. Training data DB 1702 is information for managing the training data used to train the detection model.
[0116] The learning application 1700 performs training and distribution of the detection model. The learning application 1700 includes a detection model generation unit 1710, a training data registration unit 1711, and a distribution unit 1712.
[0117] The detection model generation unit 1710 executes the detection model training process and generates a detection model. The training data registration unit 1711 registers the training data. In this embodiment, the training data registration unit 1711 acquires false detection object data from each monitoring system 10 and stores it in the training data DB 1702. The distribution unit 1712 distributes the detection model to each monitoring system 10.
[0118] Figure 18 is a flowchart illustrating an example of the detection model generation process performed by the detection model learning device 1600 of Example 4.
[0119] If the number of falsely detected object data points accumulated in the training data DB1702 exceeds a threshold, or periodically, the detection model generation process described below is executed. Note that the triggers for executing the process are not limited to those mentioned above. For example, the detection model generation process may be executed when an execution request is received from the user.
[0120] The detection model learning device 1600 performs the learning process of the detection model using the learning data DB 1702 (step S501). The learning process can be carried out using known techniques.
[0121] The detection model learning device 1600 distributes the generated detection models to each monitoring system 10 (step S502) and terminates the process.
[0122] The detection accuracy of the detection model can be improved by retraining it using the detection results from the monitoring system 10.
[0123] The monitoring system 10 may also include a detection model learning device 1600. Furthermore, the verification device 100 may have the same functionality as the detection model learning device 1600.
[0124] Representative aspects of the invention other than those described in the claims include the following: (1) A program to be executed by a computer, The computer maintains a detection model for detecting objects contained in an image, The first step is to obtain images of the space where the work is performed, A second step involves using the aforementioned detection model to detect objects contained in the image, A third step involves performing an object tracking process that tracks an object using the detection results of multiple images input in chronological order, and calculating a detection duration that represents the time during which the object detected in the latest image has been continuously detected. A fourth step involves identifying objects whose detection duration is greater than a threshold as objects to be examined, A program characterized by causing a computer to perform a fifth step of executing a verification process that determines whether or not the object to be verified is the object to be detected in the said business, based on a determination rule that takes into account the content of the said business. (2) The program described in (1), The third step described above is: A procedure for detecting an object included in the image acquired when the environment of the aforementioned space changes, A procedure for registering the detected object in the background object list, A procedure for identifying objects from among the objects detected in the latest image that are not registered in the background object list, A program characterized by including a procedure for calculating the detection duration of the identified object. (3)(2) The program described above, The program is characterized in that the third step includes a step of starting the object tracking process after a change in the spatial environment. (4)(2) The program described above, The aforementioned computer stores information about the detection target area in the image, The fifth step of the program is characterized by including a step of determining that the object to be verified, which is included in the detection target area, is the object to be detected. (5)(2) The program described above, The aforementioned computer stores information about a reference point in the image, The above step 5 is, A procedure for identifying the location of the object to be verified based on the results of the object tracking process, A program characterized by including a step of determining that the object to be verified is the object to be detected if the distance between the appearance location and the reference point is less than a threshold. (6)(2) The program described above, The fifth step described above is: A procedure for obtaining information representing the temperature of the aforementioned space, A program characterized by including a step of determining whether the object to be verified is the object to be detected, based on a comparison result of the temperature and threshold of the object to be verified. (7)(1) The program described above, The computer holds training data for training the detection model, Procedure for recording the results of the aforementioned verification, A program characterized by causing a computer to execute a procedure for performing a learning process on the detection model using the aforementioned learning data and the results of the verification.
[0125] The verification processes described in Examples 1 to 3 may be combined. Furthermore, users may be allowed to select which verification processes to use. For example, in normal operation, one of the verification processes from Examples 1 to 3 may be executed, and if a detailed analysis is required, two or more verification processes from Examples 1 to 3 may be combined and executed.
[0126] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. Furthermore, for example, the embodiments described above are detailed explanations of the configuration in order to clearly illustrate the present invention, and are not necessarily limited to those having all the configurations described. In addition, some of the configurations in each embodiment can be added to, deleted from, or replaced with other configurations.
[0127] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.
[0128] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, Python, and Java (registered trademark).
[0129] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.
[0130] In the above-described embodiment, the control lines and information lines shown are those deemed necessary for explanation and do not necessarily represent all control lines and information lines in the actual product. All components may be interconnected. [Explanation of Symbols]
[0131] 10 Monitoring System 11 Surveillance Area 12 Detection target area 100 Verification device 101 Recording device 102 Visualization device 103 Input / Output Devices 104 Camera 105, 1601 Network 200 calculator 201 CPU 202 memory 203 Network Interface 204 I / O interfaces 300 Object Detection Applications 301 Detection Model Information 302 Background object information 303 Processing Result Information 310 Image acquisition unit 311 Object detection unit 312 Event Detection Unit 313 Background Object Registration Section 314 Object Tracking Unit 315 Unattended Object Detection Unit 316 Verification Department 317 Output section 400 recording applications 401 Image Database 410 Image recording unit 411 Image Playback Unit 412 Image output section 500 Visualization Applications 501 Processing Result Information 510 Image Retrieval Unit 511 Image Generation Unit 512 Processing Result Recording Unit 1100 screens 1500 thermal imaging cameras 1600 Detection Model Learning Device 1700 Learning Applications 1701 Detection Model Information 1702 Training Data Database 1710 Detection Model Generation Unit 1711 Learning Data Registration Section 1712 Distribution Department
Claims
1. A computer having an arithmetic unit and a storage device connected to the arithmetic unit, The aforementioned storage device stores a detection model for detecting objects contained in an image, The aforementioned computing device is We take images of the space where the work is performed, Using the aforementioned detection model, objects included in the image are detected. The occurrence of an event representing a change in the environment of the aforementioned space is detected, When the environment of the aforementioned space changes, objects included in the image are detected. The detected object is registered in the background object list. After the occurrence of the aforementioned event, an object tracking process is performed to track the object using the detection results of multiple images input in chronological order, and from the objects detected in the most recent image, an object not registered in the background object list is identified. For the identified object, the detection duration, which represents the time during which it was continuously detected, is calculated. Objects whose detection duration is greater than the threshold are identified as objects to be examined. A computer characterized by performing a verification process to determine whether or not the object to be verified is the object to be detected in the said business, based on a determination rule that takes into account the content of the said business.
2. The computer according to Claim 1, The aforementioned storage device stores information about the area to be detected in the image, The computing device is characterized in that, in the verification process, it determines that the object to be verified, which is included in the detection target area, is the detection target object.
3. The computer according to Claim 1, The aforementioned storage device stores information about the reference point in the image, The aforementioned computing device, in the verification process, Based on the results of the object tracking process, the location where the object to be verified appears is identified. A computer characterized by determining that the object to be verified is the object to be detected if the distance between the appearance location and the reference point is less than a threshold.
4. The computer according to claim 1, The aforementioned computing device is Obtain information representing the temperature of the aforementioned space, The verification process is characterized by determining whether the object to be verified is the object to be detected, based on the results of comparing the temperature and threshold of the object to be verified.
5. A computer according to claim 1, The memory device stores training data for training the detection model, The aforementioned computing device is The results of the verification process are recorded, A computer characterized by performing a learning process for the detection model using the aforementioned learning data and the results of the verification process.
6. A computer-based object detection method, The computer has an arithmetic unit and a storage device connected to the arithmetic unit. The aforementioned storage device stores a detection model for detecting objects contained in an image, The object detection method described above is: The first step is for the computing device to acquire an image of the space where the work is performed, The calculation device performs a second step of detecting objects included in the image using the detection model, The calculation device performs a third step of detecting the occurrence of an event that represents a change in the environment of the space, The calculation device detects objects included in the image acquired when the environment of the space changes, and registers the detected objects in the background object list, A fifth step in which the computing device performs an object tracking process to track an object using the detection results of multiple images input in chronological order after the occurrence of the event, The calculation device performs a sixth step in which it identifies objects from among the objects detected in the latest image that are not registered in the background object list, and calculates a detection duration representing the time during which the identified object has been continuously detected. The seventh step is for the calculation device to identify an object whose detection duration is greater than a threshold as an object to be examined, The eighth step is for the computing device to perform a verification process that determines whether the object to be verified is a detection target object in the business, based on a determination rule that takes into account the content of the business, An object detection method characterized by including the following.
7. The object detection method according to claim 6, The aforementioned storage device stores information about the area to be detected in the image, The object detection method is characterized in that the eighth step includes the step of the computing device determining that the object to be verified, which is included in the detection target area, is the detection target object.
8. The object detection method according to claim 6, The aforementioned storage device stores information about the reference point in the image, The eighth step described above is: The calculation device performs the steps of determining the appearance location of the object to be verified based on the results of the object tracking process, An object detection method characterized in that the calculation device determines the object to be verified as the object to be detected if the distance between the appearance position and the reference point is less than a threshold.
9. The object detection method according to claim 6, The eighth step described above is: The calculation device obtains information representing the temperature of the space, An object detection method characterized by comprising the step of the computing device determining whether or not the object to be verified is the object to be detected, based on the result of comparing the temperature and threshold of the object to be verified.
10. The object detection method according to claim 6, The memory device stores training data for training the detection model, The object detection method described above is: The arithmetic unit records the results of the verification process, An object detection method characterized by comprising the step of the computing device performing a learning process for the detection model using the learning data and the results of the verification process.
11. A system, An imaging device that photographs the space where work is performed, A recording device that acquires and records images from the aforementioned imaging device, A verification device that acquires an image from the aforementioned imaging device and detects objects contained in the image, The device comprises a recording device for playing back images recorded in the recording device and a visualization device for displaying the processing results of the verification device, The verification device comprises a computing device and a storage device connected to the computing device. The storage device stores a detection model for detecting objects included in the image, The aforementioned computing device is Images of the space where the aforementioned work is performed are taken. Using the aforementioned detection model, objects included in the image are detected. The occurrence of an event representing a change in the environment of the aforementioned space is detected, When the environment of the aforementioned space changes, objects included in the image are detected. The detected object is registered in the background object list. After the occurrence of the aforementioned event, an object tracking process is performed to track the object using the detection results of multiple images input in chronological order, and from the objects detected in the most recent image, an object not registered in the background object list is identified. For the identified object, the detection duration, which represents the time during which it was continuously detected, is calculated. Objects whose detection duration is greater than the threshold are identified as objects to be examined. A system characterized by performing a verification process to determine whether or not the object to be verified is the object to be detected in the said business, based on a determination rule that takes into account the content of the said business.
12. The system according to claim 11, The device includes a learning device for learning the aforementioned detection model, The learning device is The system holds training data for training the aforementioned detection model, The results of the verification process are obtained from the verification device. Using the aforementioned training data and the results of the verification process, the training process of the detection model is executed. A system characterized by transmitting the new detection model generated by the learning process to the verification device.