Object tracking across multiple images

Abstract

The system includes a processor and a memory storing instructions that, when executed by the processor, configure the processor to receive a plurality of images captured by a camera from an image processing system, detect an object in an image from the plurality of images using a model, and identify the detected object using the model and a database of previously identified objects. The instructions configure the processor to track movement of the identified object across a sequence of images from the plurality of images. The instructions configure the processor to detect when the identified object disappears from the view of the camera based on the plurality of images. The instructions configure the processor to determine a result related to the identified object based on the first and last detection of the identified object and the direction of movement of the identified object.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 173,265, filed on April 9, 2021. The entire disclosure of the application mentioned above is incorporated herein by reference.

[0002] This disclosure generally relates to detecting objects in multiple images, and more particularly to tracking the movement of objects across those images.

Background Art

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. Within the scope described in this background art section, the achievements of the inventors whose names are currently cited, as well as aspects of this description that may not be eligible as prior art in other respects at the time of filing, are not to be recognized, either explicitly or implicitly, as prior art against the present disclosure.

[0004] Various technologies are used to capture images of objects. For example, in the medical field, ultrasound, magnetic resonance imaging (MRI), computed tomography (CT) scans, and other technologies are used to capture images of organs. Surgical procedures are monitored, and images are captured using cameras. In retail applications, cameras can be used to capture images of items on store shelves or stored in warehouses to monitor inventory levels and replenishment. In the automotive industry, cameras mounted on or inside vehicles capture images of objects around the vehicle. In security systems, cameras monitor and record images of areas inside and around buildings. In astronomical and space-related applications, cameras capture images of celestial bodies. In hazardous applications, cameras monitor processes involving hazardous materials and / or operations, and capture images of those processes. In robotics, robots operate based on images captured by cameras, and so on. [Overview of the project] [Means for solving the problem]

[0005] A system comprising a processor and memory storing instructions, wherein the processor is configured to perform the following actions when executed by the processor: receive multiple images captured by a camera from an image processing system; detect an object in one of the multiple images using a model; and identify the detected object using the model and a database of previously identified objects. The instructions configure the processor to track the movement of the identified object across a series of images from the multiple images. The instructions configure the processor to detect, based on the multiple images, when the identified object disappears from the camera's view. The instructions configure the processor to determine results regarding the identified object based on the first and last detections of the identified object and the direction of the identified object's movement.

[0006] In another feature, identifying the outcome includes determining whether the identified object remains within the observed area or whether the identified object has left the observed area.

[0007] In other features, these instructions configure the processor to assign a timestamp to each instance of the detected object, assign a label to the detected object, and assign bounding box coordinates to the detected object. These instructions configure the processor to store the timestamp, label, and bounding box coordinates in the detection history for each instance of the detected object. These instructions configure the processor to determine the first and last detections of the detected object, as well as the direction of movement of the detected object, based on the detection history for the detected object.

[0008] In other features, these instructions configure the processor to assign a certainty score to the label for each instance of detecting an identified object, and to increase the certainty score with each subsequent detection of an identified object.

[0009] In another feature, these instructions configure the processor to predict, with increased certainty, the detection of subsequent objects in the direction of movement.

[0010] In other features, the instructions configure the processor to detect an identified object in N1 images from a plurality of images, where N1 is an integer greater than 1. The instructions also configure the processor to determine that if the identified object disappears after N1 images but reappears in N2 or fewer images from a plurality of images following N1 images, the detected identified object in N2 images is a continuation of the detected identified object in N1 images.

[0011] In another feature, these instructions configure the processor to determine that an identified object is outside the camera's view if it is not detected in N1+N2 of the images.

[0012] In other features, the system includes a processor and memory storing instructions, which, when executed by the processor, configure the processor to receive images captured by first and second cameras from an image processing system, detect objects in the images using a model, and identify the detected objects using the model and a database of previously identified objects. These instructions configure the processor to track the movement of identified objects across multiple images by correlating multiple detections of identified objects in the images.

[0013] In another feature, these instructions configure the processor to detect, with increased certainty, the identified object in one of the images from the first camera in response to the detection of the identified object in one of the images from the second camera.

[0014] In other features, the instructions configure the processor to detect that the identified object is moving in the same direction across multiple images from the first and second cameras. In response to detecting that the identified object is moving in the same direction across multiple images from the first and second cameras, the instructions configure the processor to track the movement of the identified object with enhanced certainty.

[0015] In another feature, these instructions configure the processor to predict, with increased certainty, the detection of subsequent objects in the direction of movement.

[0016] In other features, these instructions configure the processor to detect when the identified object disappears from the view of the first camera. In response to the identified object disappearing from the view of the first camera, these instructions configure the processor to track the movement of the identified object in multiple images from the second camera.

[0017] In other features, these instructions configure the processor to detect when the identified object disappears from the view of the second camera. These instructions configure the processor to determine the first and last detection of the identified object in the images from the first and second cameras, as well as the direction of movement of the identified object. These instructions configure the processor to determine results regarding the identified object based on the first and last detection of the identified object and the direction of movement of the identified object.

[0018] In another feature, identifying the outcome includes determining whether the identified object remains within the observed area or whether the identified object has left the observed area.

[0019] In other features, these instructions configure the processor to assign a timestamp to each instance of the detection of an identified object, assign a label to the identified object, and assign bounding box coordinates to the identified object. These instructions configure the processor to store the timestamp, label, and bounding box coordinates in the detection history for each instance of the detection of an identified object. These instructions configure the processor to identify the first and last detections of the identified object, as well as the direction of movement of the identified object, by correlating the detection history for the identified object.

[0020] In other features, these instructions configure the processor to assign a timestamp to the detection of the identified object, assign a label to the identified object, and assign bounding box coordinates to the identified object for each instance of detection of the identified object. These instructions configure the processor to store the timestamp, label, and bounding box coordinates in the detection history for the identified object for each instance of detection of the identified object. These instructions configure the processor to track the movement of the identified object across multiple images by correlating the detection history for the identified object.

[0021] Further scope of application of this disclosure will become apparent from the detailed description, claims, and drawings. The detailed description and specific examples are intended for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0022] This disclosure will be better understood from the detailed description and accompanying drawings. [Brief explanation of the drawing]

[0023] [Figure 1]A diagram showing a system for detecting an object in an image and tracking the movement of the object across a plurality of images according to the present disclosure. [Figure 2] A diagram showing a system for detecting an object in an image and tracking the movement of the object across a plurality of images according to the present disclosure. [Figure 3] A diagram showing a method for detecting an object in an image and tracking the movement of the object across a plurality of images according to the present disclosure. [Figure 4A] A diagram showing a method for detecting an object in an image, tracking the object across a plurality of images, and specifying the direction of movement of the object according to the present disclosure. [Figure 4B] A diagram showing an example of specifying the direction of movement regarding a tracked object using bounding box coordinates. [Figure 5] A diagram showing a method for specifying when a tracked object goes out of the view according to the present disclosure. [Figure 6] A diagram showing a method for specifying the result when a tracked object goes out of the view according to the present disclosure. [Figure 7] A diagram showing a method for detecting and tracking an object across a plurality of images received from a plurality of sources (e.g., a plurality of cameras) according to the present disclosure. [Figure 8] A diagram showing a method for detecting and tracking an object across a plurality of images received from a plurality of sources, and specifying the direction of movement of the object with enhanced certainty according to the present disclosure. [Figure 9] A diagram showing a method for detecting and tracking an object across a plurality of images received from a plurality of cameras when the object goes out of the view of one camera but then appears in the view of another camera according to the present disclosure. [Figure 10] A diagram showing a method for detecting and tracking an object across a plurality of images received from two cameras when the view of one camera is blocked while the other camera continues to see the object according to the present disclosure.

[0024] In drawings, reference numbers may be reused to identify similar and / or identical elements. [Modes for carrying out the invention]

[0025] This disclosure relates to detecting objects in images captured using various techniques and tracking the movement of detected objects across a series of images. Specifically, a machine learning-based model processes images to identify and locate objects captured in those images. When an object that the model has been trained to recognize appears in the image input to the model, the model assigns an identification label to the object, assigns a certainty score for the assigned identification label, and provides a set of coordinates representing the object's location in the image, which are used to construct a bounding box around the object in the image. The bounding box is used to track the object. When a series of images has been processed by the model and objects have been detected consecutively across multiple images, the bounding box can be used to track the object's movement across the series of images, as described in detail below.

[0026] When a first image containing an object trained to recognize the model is processed by the model, the system begins tracking the identified object by initiating the structure of the object to be tracked. The structure of the object to be tracked has several properties, including an object label that indicates the type of object assigned by the model, a history of detections assigned to the object by the model, and a unique identifier assigned by the model to each instance of object detection to distinguish between individual object detections, even if multiple detected objects are of the same type. When the object to be tracked is first detected, an identification label and a new object identifier are assigned to the object to be tracked, and a first bounding box is stored in the detection history of the object to be tracked.

[0027] When subsequent images are processed by the model and the model makes a new detection of a previously detected object (i.e., the tracked object), the system compares the new detection to the existing tracked object. In a new detection of a tracked object, the tracked object is assigned a higher certainty score than the certainty scores assigned to previous detections of that object. If a new detection of a tracked object occurs within a predetermined range from the last recorded detection of the tracked object, the new detection is considered a continuation of the tracked object and is added to the detection history of the tracked object. If multiple detections of a tracked object fall within a predetermined range for a single object, the nearest detection of the tracked object is considered a continuation of that object, or, if the tracking history has sufficient data, it is possible to use that history to determine the approximate direction of the tracked object's movement. The approximate direction can be used as a predictor to prioritize new detections of the tracked object in that direction.

[0028] If a tracked object is not detected within a predetermined number of images, it is considered to be out of view (i.e., has disappeared from view), and the tracking system is prompted to make a decision about the outcome of the object's detection or movement. Depending on the use case, other criteria may be used to prompt the decision, such as the object reaching a specific segment of the image. Once an object is considered out of view and the tracking system is prompted to make a decision about it, the outcome regarding the object is determined using the object's detection history and identification information. Observations that can be extracted and used from the object's detection history include the origin (i.e., where the object was when it was first detected), the point of departure (i.e., where the object was when it was lost from view), and the approximate direction in which the object was moving when it was lost from view.

[0029] In some applications, it is possible to process images from multiple sources (e.g., captured by separate cameras) using a model, and to correlate multiple detections from multiple sources to improve object detection and tracking. For example, images from two different sources can provide different views of the same object. When the same object is detected in images from multiple sources, the identification information for that object can be assigned a higher certainty score than when the object is detected in an image from a single source. Furthermore, it is possible to improve object tracking using images from multiple sources. For example, object tracking performed using images from multiple sources can determine the direction in which the tracked object is moving. Additionally, even if the tracked object may move out of view in an image from one source, it may still remain in view in an image from another source, and therefore can be further tracked. If the tracked object eventually does move out of view, object tracking using multiple sources can provide better results for the tracked object than identification based on an image from a single source.

[0030] In addition, when an item is tracked using multiple sources, the model can use timestamps to mark when the item was first detected by a particular source and when it was last updated by a particular source. This allows for further complexity and redundancy in the decision-making process. This is useful, for example, in cases where the view of an item is obstructed by one source, but another source continues to observe it.

[0031] For example, suppose an item enters the views of both camera 1 and camera 2 at the start of the tracking process. A timestamp is taken when the item is first detected by both cameras. Subsequently, a failure occurs during the tracking process, so that the item is no longer visible to camera 1, but is still visible to camera 2. The model retains a timestamp of the last time camera 1 saw the item. The item continues to be tracked by camera 2 until it leaves the view of camera 2 or otherwise triggers a decision. Although the item is lost from camera 1, it is still being tracked by camera 2. Therefore, the model can wait until the item leaves the view of the last camera (i.e., camera 2 in this example). The item then leaves the area being observed, and now camera 2 also loses its view of the item. The model is then prompted to make a decision about the tracked item.

[0032] As part of the decision-making process, the model compares the timestamps of when the item was first detected by both cameras with the timestamps of when the item was last detected by both cameras to determine which camera has the oldest information about the item's origin and which camera has the most recent information about where the item went. In this example, cameras 1 and 2 detected the item simultaneously, but due to a failure, camera 1 lost sight of the item, and then camera 2 lost sight of the item. The model uses the information provided by camera 2 to identify the item's departure point and determine the outcome regarding the item.

[0033] The systems and methods of this disclosure can be bundled as turnkey solutions customized for specific applications. Alternatively, at least some parts of the systems and methods, such as models, can be implemented as Software-as-a-Service (SaaS). The SaaS portion can be hosted in the cloud, interfaced with a local image acquisition system, and supplied on a subscription basis. These and other features of this disclosure are described in further detail here.

[0034] This disclosure is organized as follows: A system for detecting an object in an image and tracking the movement of an object across multiple images is shown and described with reference to Figures 1 and 2. A method for detecting an object in an image and tracking the movement of an object across multiple images is shown and described with reference to Figure 3. A method for detecting an object in an image, tracking an object across multiple images, and determining the direction of movement of an object is shown and described with reference to Figure 4A. An example of determining the direction of movement of a tracked object using bounding box coordinates is shown and described with reference to Figure 4B. A method for determining when a tracked object has moved out of view is shown and described with reference to Figure 5. A method for determining the result when a tracked object has moved out of view is shown and described with reference to Figure 6.

[0035] Subsequently, methods for detecting and tracking an object across multiple images received from multiple sources (e.g., multiple cameras) are shown and described with reference to Figure 7. Methods for detecting and tracking an object across multiple images received from multiple sources and determining the direction of the object's movement with increased certainty are shown and described with reference to Figure 8. Methods for detecting and tracking an object across multiple images received from multiple cameras when the object moves out of the view of one camera but then reappears in the view of another camera are shown and described with reference to Figure 9. Methods for detecting and tracking an object across multiple images received from two cameras when the view of one camera is obstructed while the other camera continues to view the object are shown and described with reference to Figure 10.

[0036] Figure 1 shows a system 100 according to the present disclosure for detecting an object in an image and tracking the movement of an object across multiple images. The system 100 includes an image acquisition system 102, an image processing system 104, an object detection system 106, and an object tracking system 108. For example, the image acquisition system 102 includes one or more image acquisition devices, such as a camera. For example, the image processing system 104 processes the image acquired by the image acquisition system 102 by filtering noise from the image, by adjusting attributes such as color, brightness, and contrast of the image.

[0037] The object detection system 106 detects one or more objects in the processed image received from the image processing system 104 according to this disclosure, as described in more detail thereafter. The object tracking system 108 tracks one or more objects detected by the object detection system 106 according to this disclosure, as described in more detail thereafter. The object detection system 106 is shown in more detail in Figure 2.

[0038] Each of the object detection system 106 and the object tracking system 108 includes one or more processors and memory that perform one or more methods described hereafter with reference to Figures 3 to 10. In other words, each of the object detection system 106 and the object tracking system 108 can be implemented as including one or more modules, which are defined hereafter following the descriptions of the systems and methods of this disclosure.

[0039] In some examples, the object detection system 106 and the object tracking system 108 can be integrated into a single system (for example, a single module). In some examples, the object detection system 106 can also perform object tracking. In some examples, the object tracking system 108 can perform result identification after the tracked object has moved out of view, as described in detail below.

[0040] Figure 2 illustrates the object detection system 106 according to this disclosure in more detail. For example, the object detection system 106 includes an object detection model 120, an object database 122, and a history database 124. The object detection model 120 is trained (for example, using machine learning) to detect objects in images received from the image processing system 104. For example, the object detection model 120 is trained to identify objects based on one or more views of objects captured by one or more cameras of the image acquisition system 102. Views (i.e., images) captured from various angles by one or more cameras are stored in the object database 122 after processing by the image processing system 104. The object database 122 stores many views of various objects that the object detection model 120 is trained to detect.

[0041] When in use, the object detection model 120 detects objects in images received from the image processing system 104 using the method described in detail below. If an object is detected, the object detection model 120 compares the detected object to one or more objects stored in the object database 122. If an object matching the detected object is found in the object database 122, the object detection model 120 assigns various parameters to the detected object. These parameters include an identification label for the detected object, a certainty score for the assigned identification label, an identifier for the instance of object detection, and bounding box coordinates for the detected object, the bounding box coordinates indicating the location of the detected object in the image. These parameters, along with a timestamp for the instance of object detection, are stored in the detection history of the detected object in the history database 124. Using the detection history, the object detection model 120 tracks the detected object and determines the direction of movement of the tracked object using the method described in detail below.

[0042] As described above, system 100 can be supplied as a turnkey solution customized for a specific application (for example, for retail or medical use). Alternatively, at least one of the image processing system 104, object detection system 106, and object tracking system 108 can be implemented in the cloud as Software-as-a-Service (SaaS), interfaced with the locally deployed image acquisition system 102, and supplied on a subscription basis.

[0043] Figure 3 illustrates a method 150 according to this disclosure for detecting an object in an image and tracking the movement of an object across multiple images. Method 150 is shown and described in general with reference to Figure 3 and more specifically (i.e., in detail) with reference to Figures 4 to 6. Method 150 is performed in part by an object detection model 120 of an object detection system 106 and in part by an object tracking system 108.

[0044] In step 152, the object detection model 120 identifies objects in images received from the image processing system 104. Object detection is described in more detail later with reference to Figure 4. In step 154, the object detection model 120 tracks the movement of objects by detecting them in a series of images received from the image processing system 104. The object detection model 120 maintains a history of the detection of the tracked object in the history database 124. Object tracking is described in more detail later with reference to Figures 4 and 5.

[0045] In step 156, the object tracking system 108 determines whether the object being tracked is outside the view, as described below with reference to Figure 5. If the object being tracked is not outside the view, method 150 returns to step 154 ​​(i.e., the object detection model 120 continues to track the object). If the object being tracked is outside the view, in step 158, the object tracking system 108 determines the outcome regarding the object being tracked (for example, what happened to the object being tracked), as described below in detail with reference to Figure 6.

[0046] Figure 4A further illustrates the method 200 provided in this disclosure for detecting an object in an image, tracking an object across multiple images, and determining the direction of movement of an object. In 202, the object detection model 120 receives an image from a source. In 204, the object detection model 120 detects an object in the first image. In 206, the object detection model 120 assigns an identifier to the detection. In 208, the object detection model 120 identifies the detected object using the object database 122 (i.e., by comparing the detected object to an object in the object database 122). In 210, the object detection model 120 assigns a label to the identified object. In 212, the object detection model 120 assigns a certainty score to the label assigned to the identified object. In 214, the object detection model 120 provides bounding box coordinates for the identified object (shown in detail with reference to Figure 4B). In 216, the object detection model 120 stores the detection data (i.e., identifier, label, certainty score, and bounding box coordinates) in the history database 124, which is maintained for identified objects.

[0047] In step 218, the object detection model 120 detects an object in the next image. In step 220, the object detection model 120 assigns an identifier to the detection. In step 222, the object detection model 120 identifies the detected object using the object database 122. In step 224, the object detection model 120 assigns a label to the identified object (i.e., by comparing the detected object with an object in the object database 122). In step 226, the object detection model 120 determines whether the object identified in the next image is the same object identified in the previous image. If the object identified in the next image is not the same object identified in the previous image, method 200 returns to step 212.

[0048] If the object identified in the next image is the same object identified in the previous image, in 228, the object detection model 120 assigns a higher certainty score to the label assigned to the identified object in the next image. In 230, the object detection model 120 provides bounding box coordinates for the identified object in the next image (shown in detail with reference to Figure 4B). In 232, the object detection model 120 adds the detection data (i.e., identifier, label, certainty score, and bounding box coordinates) to the history held for the identified object in the history database 124.

[0049] In step 234, the object tracking system 108 determines the direction of movement of the identified object based on the bounding box coordinates stored in the history of the identified object. For example, the object tracking system 108 determines the direction of movement of the tracked object by subtracting the bounding box coordinates in the first instance of detection of the tracked object from the bounding box coordinates in the last instance of detection of the tracked object.

[0050] Figure 4B illustrates an example of using bounding box coordinates to determine the direction of motion for a tracked object. In the example shown, object 242 is tracked in a series of images 240-1, 240-2, ..., and 240-N (collectively referred to as image 240), where N is a positive integer. In each image 240, a bounding box 244 is assigned to object 242 when it is detected and identified as described above. Each bounding box 244 has coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4). The direction of motion for object 242, which can be calculated between any two images 240-i and 240-j, can be calculated by subtracting the coordinates of bounding box 244-j from the coordinates of bounding box 244-i.

[0051] For example, the direction 246-1 of the movement of object 242 between images 240-1 and 240-2 is the difference between the coordinates of bounding boxes 244-2 and 244-1. The direction 246-2 of the movement of object 242 between images 240-2 and 240-N is the difference between the coordinates of bounding boxes 244-N and 244-2. If object 242 is first detected in image 240-1 and last detected in image 240-N, the net direction 246-3 of the movement of object 242 between images 240-1 and 240-N is the difference between the coordinates of bounding boxes 244-N and 244-1.

[0052] Figure 5 illustrates method 250 according to the present disclosure for determining when a tracked object has moved out of view. In 252, the object detection model 120 receives images from the source. In 254, the object detection model 120 performs object detection on a series of images as described above with reference to Figure 4. In 256, the object detection model 120 determines whether the identified object was not detected in N1 consecutive images, where N1 is a positive integer. Method 250 returns to 254, and if the identified object is still detected, the object detection model 120 continues to perform object detection.

[0053] If the identified object is not detected in N1 consecutive images, at 258, the object detection model 120 determines whether the identified object was not detected in a further N2 consecutive images following the N1 images (i.e., whether the identified object was not detected in N1+N2 consecutive images), where N2 is a positive integer. If the identified object is detected in a further N2 consecutive images (i.e., the identified object was not detected in N1 images but was detected in the N2 images following the N1 images), at 260, the object detection model 120 determines that the detection is a continuation of the same object identified in images prior to the N1 images (i.e., the identified object is the same object), and method 250 returns to 254, where the object detection model 120 continues to perform object detection.

[0054] If the identified object is not detected in N2 consecutive images following N1 images (i.e., if the identified object is not detected in N1+N2 consecutive images), then in 262, the object detection model 120 determines that the identified object is outside the view. In 264, the object tracking system 108 determines the result regarding the tracked object (e.g., what happened to the tracked object), as described in detail thereafter with reference to Figure 6.

[0055] Figure 6 illustrates a method 280 according to this disclosure for determining the consequences when a tracked object moves out of view. In 282, the object tracking system 108 retrieves detection data stored in the history of the tracked object in the history database 124. In 284, the object tracking system 108 locates the origin of the tracked object (i.e., the first detection instance) (for example, based on the bounding box coordinates provided by the object detection model 120 in the first detection instance of the tracked object). In 286, the object tracking system 108 locates the point of departure of the tracked object (i.e., the last detection instance) (for example, based on the bounding box coordinates provided by the object detection model 120 in the last detection instance of the tracked object).

[0056] In 288, the object tracking system 108 determines the direction of movement of the tracked object based on the origin and departure points identified with respect to the tracked object. For example, the object tracking system 108 determines the direction of movement of the tracked object by subtracting the bounding box coordinates in the first instance of detection of the tracked object from the bounding box coordinates in the last instance of detection of the tracked object.

[0057] In 290, the object tracking system 108 identifies the outcome concerning the tracked object (i.e., what happened to the tracked object when it moved out of view) based on the identified direction of the tracked object's movement. For example, the object tracking system 108 identifies whether the tracked object entered or left the area being observed. For example, the area being observed could be a storage shelf being monitored (e.g., a cabinet, refrigerator, etc.), and the tracked object could be food (e.g., a milk carton). For example, the area being observed could be a surgical area being monitored (e.g., the abdomen), and the tracked object could be a surgical instrument (e.g., a scalpel). For example, the area being observed could be a region in outer space being monitored, and the tracked object could be a celestial body (e.g., a star, planet, etc.).

[0058] Figure 7 shows a method 300 for detecting and tracking an object across multiple images received from multiple sources (e.g., multiple cameras) according to the present disclosure. In 302, the object detection model 120 receives an image from a first source (e.g., an image captured by a first camera of an image acquisition system 102 and processed by an image processing system 104). In 304, the object detection model 120 receives an image from a second source (e.g., an image captured by a second camera of an image acquisition system 102 and processed by an image processing system 104). In 306, the image detection model 120 performs object detection based on the images received from the first and second sources, as will be described in more detail later with reference to Figure 8. In 308, the image detection model 120 improves object detection and tracking by correlating the multiple object detections performed based on the images received from the first and second sources, as will be described in more detail later with reference to Figures 8 to 10.

[0059] Figure 8 illustrates a method 350 according to the present disclosure for detecting and tracking an object across multiple images received from multiple sources and determining the direction of the object's movement with enhanced certainty. In 352, the object detection model 120 receives an image from a first source (e.g., an image captured by a first camera of an image acquisition system 102 and processed by an image processing system 104). In 354, the object detection model 120 receives an image from a second source (e.g., an image captured by a second camera of an image acquisition system 102 and processed by an image processing system 104).

[0060] In step 356, the object detection model 120 detects an object in the first image received from the first source, as described above with reference to Figures 4 and 5. In step 358, the object detection model 120 detects an object in the second image received from the second source, as described above with reference to Figures 4 and 5. In step 360, the object detection model 120 determines whether the multiple objects detected in the first and second images are the same object. If the multiple objects detected in the first and second images are not the same object, method 350 returns to step 352, and the object detection model 120 continues to detect objects in the images received from the first and second sources, as described above with reference to Figures 4 and 5. If multiple objects detected in the first and second images are the same object, the object detection model 120 assigns a higher certainty score to the label assigned to the object detected in the second image than the certainty score assigned to the label previously assigned to the object detected in the first image.

[0061] In step 364, the object detection model 120 identifies the direction of movement of the objects detected in the first and second images, as described above with reference to Figure 6. The object detection model 120 determines whether the direction of movement of multiple objects detected in the first and second images is the same. If the direction of movement of multiple objects detected in the first and second images is the same, in step 366, the object detection model 120 indicates increased certainty in the direction of movement of the objects detected in the first and second images, and method 350 ends. In addition, the object detection model 120 uses that direction to predict subsequent detection of objects in that direction with increased certainty. That is, the object detection model 120 can use that direction as a predictor to prioritize new detections of tracked objects in that direction.

[0062] If the directions of movement of multiple objects detected in the first and second images are not the same, in 368, the object detection model 120 proposes or initiates a corrective action, and method 350 terminates. For example, the corrective action may include generating an alert that the same object seen by the two cameras is moving in a diverging direction, which may indicate a problem in system 100 (for example, one or more elements of system 100 may need to be updated).

[0063] Figure 9 illustrates a method 400 for detecting and tracking an object across multiple images received from multiple cameras when the object moves out of the view of one camera but subsequently reappears in the view of another camera. In 402, the object detection model 120 receives an image from a first source (e.g., an image captured by a first camera of the image acquisition system 102 and processed by the image processing system 104). In 404, the object detection model 120 receives an image from a second source (e.g., an image captured by a second camera of the image acquisition system 102 and processed by the image processing system 104). In 406, the object detection model 120 detects an object in the first image received from the first source, as described above with reference to Figures 4 and 5.

[0064] In step 408, the object detection model 120 determines from the image received from the first source whether the object has moved out of view, as described above with reference to Figure 5. If the object has not moved out of view, method 400 returns to step 406, where the object detection model 120 continues to detect and track the object based on the image received from the first source.

[0065] If the object is outside the view in the image received from the first source, in 410, the object detection model 120 detects the object in the second image received from the second source, as described above with reference to Figures 4 and 5. In 412, the object detection model 120 determines whether the multiple objects detected in the first and second images from the first and second sources are the same object. If the multiple objects detected in the first and second images are not the same object, method 400 returns to 402, and the object detection model 120 continues to detect objects in the images received from the first and second sources, as described above with reference to Figures 4 and 5. If the multiple objects detected in the first and second images are the same object, the object detection model 120 continues to detect and track the object in the image received from the second source, as described above with reference to Figures 4 and 5.

[0066] In step 416, the object detection model 120 determines from the image received from the second source whether the object has moved out of view, as described above with reference to Figure 5. If the object has not moved out of view, method 400 returns to 414, and the object detection model 120 continues to detect and track the object based on the image received from the second source. If the object has moved out of view from the image received from the second source, in step 418, the object tracking system 108 retrieves the history of object detection in the images received from both the first and second sources from the history database 124.

[0067] In 420, the object tracking system 108 locates the origin of the object being tracked (i.e., the first instance of detection in the images received from the first and second sources). For example, the object tracking system 108 locates the origin based on the bounding box coordinates provided by the object detection model 120 in the first instance of detection of the object being tracked in the images received from the first and second sources.

[0068] In 422, the object tracking system 108 locates the point of departure of the tracked object (i.e., the last detection instance in the images received from the first and second sources). For example, the object tracking system 108 locates the point of departure based on the bounding box coordinates provided by the object detection model 120 in the last detection instance of the tracked object in the images received from the first and second sources.

[0069] In 424, the object tracking system 108 determines the direction of movement of the tracked object based on the origin and departure points identified with respect to the tracked object. In 426, the object tracking system 108 determines the outcome with respect to the tracked object (i.e., what happened to the tracked object when it moved out of view) based on the identified direction of movement of the tracked object. For example, the object tracking system 108 determines whether the tracked object entered or left an area observed by the first and second sources, an example of which is provided above with reference to Figure 6 and is therefore not repeated for brevity.

[0070] Figure 10 illustrates a method 450 according to the present disclosure for detecting and tracking an object across multiple images received from two cameras when the view of one camera is obstructed while the other camera continues to view the object. In 452, the object detection model 120 receives an image from a first source (e.g., an image captured by a first camera of an image acquisition system 102 and processed by an image processing system 104). In 454, the object detection model 120 receives an image from a second source (e.g., an image captured by a second camera of an image acquisition system 102 and processed by an image processing system 104).

[0071] In step 456, the object detection model 120 detects an object in a first image received from a first source, as described above with reference to Figures 4 and 5. The object detection model 120 timestamps the first view of the object by the first source (i.e., the object detection model 120 timestamps the first instance of object detection based on the image received from the first source). The timestamped detection data is stored in the history database 124 in the detection history of the tracked object.

[0072] In step 458, the object detection model 120 detects the same object in the second image received from the second source, as described above with reference to Figures 4 and 5. The object detection model 120 timestamps the first view of the object from the second source (i.e., the object detection model 120 timestamps the first instance of object detection based on the image received from the second source). The timestamped detection data is stored in the history database 124 in the detection history of the tracked object.

[0073] In step 460, the object detection model 120 continues to detect and track the object in the images received from the first and second sources, as described above with reference to Figures 4 and 5. In step 462, the object detection model 120 determines whether the object has moved out of view from the image received from the first source, as described above with reference to Figure 5. If the object has not moved out of view, method 450 returns to 460, and the object detection model 120 continues to detect and track the object based on the images received from the first and second sources, as described above with reference to Figures 4 and 5.

[0074] If the object has moved out of view from the image received from the first source, in 464, the object detection model 120 timestamps the last view of the object by the first source (i.e., the object detection model 120 timestamps the last instance of object detection based on the image received from the first source). The timestamped detection data is stored in the detection history of the tracked object in the history database 124. In 466, the object detection model 120 continues to detect and track the object based on the image received from the second source, as described above with reference to Figures 4 and 5.

[0075] In step 468, the object detection model 120 determines whether the object has moved out of view from the image received from the second source, as described above with reference to Figure 5. If the object has not moved out of view from the image received from the second source, method 450 returns to 466, and the object detection model 120 continues to detect and track the object based on the image received from the second source, as described above with reference to Figures 4 and 5. If the object has moved out of view from the image received from the second source, in step 470, the object detection model 120 timestamps the last view of the object by the second source (i.e., the object detection model 120 timestamps the last instance of object detection based on the image received from the second source). The timestamped detection data is stored in the detection history of the tracked object in the history database 124.

[0076] In step 472, based on the timestamped detection data stored in the detection history of the tracked object in the history database 124, the object tracking system 108 identifies that the detection data for the object obtained from the image from the second source is later in time than the detection data for the object obtained from the image from the first source. That is, the object tracking system 108 identifies that the detection data for the object obtained from the image from the second source is the most recent detection data for that object. Specifically, the object tracking system 108 compares the timestamps of when the object was first detected in the images from both sources with the timestamps of when the object was last detected in the images from both sources. Based on this comparison, the object tracking system 108 identifies which source's detection data contains the oldest information about the object's origin (in this example, the first source) and which source's detection data contains the most recent information about where the object went (in this example, the second source).

[0077] In step 474, the object tracking system 108 retrieves the history of detections of the tracked object in the images received from both the first and second sources from the history database 124. In step 476, the object tracking system 108 locates the origin of the tracked object (i.e., the first instance of detection of the tracked object in the images received from the first and second sources). For example, the object tracking system 108 locates the origin based on a timestamp provided by the object detection model 120 in the first instance of detection of the tracked object in the images received from the first and second sources.

[0078] In 422, the object tracking system 108 locates the point of departure of the tracked object (i.e., the last detection instance in the image received from the second source). For example, the object tracking system 108 locates the point of departure based on a timestamp provided by the object detection model 120 in the last detection instance of the tracked object in the image received from the second source, which is identified as the most recent detection data regarding the tracked object, as described above.

[0079] In 480, the object tracking system 108 determines the direction of movement of the tracked object based on the identified origin and departure points with respect to the tracked object. In 482, the object tracking system 108 determines the outcome with respect to the tracked object (i.e., what happened to the tracked object when it moved out of view) based on the identified direction of movement of the tracked object. For example, the object tracking system 108 determines whether the tracked object entered or left the area being observed, an example of which is provided above with reference to Figure 6 and is therefore not repeated for brevity.

[0080] This provides an illustrative example of detecting and tracking objects using the system and method described above. Assume that system 100 is set up to monitor items being added to and removed from a refrigerator, and that object detection model 120 is trained to recognize food. System 100 is informed that one side of the image is the refrigerator door, and all other sides are away from the refrigerator. The image is passed to object detection model 120. Object detection model 120 recognizes the item in the image as milk and begins tracking the object with the first detection passed into its history. The first detection places the bounding box coordinates away from the refrigerator. Several more images are processed. Each processed image indicates that the direction the item appears to be moving is towards the refrigerator until the milk is deemed to be out of view. System 100 is then prompted to make a decision about what happened to the item. The first detection in the object's detection history was away from the refrigerator, and the last detection before the item went out of view was very close to the refrigerator side of the image. Using some of the last detections, system 100 determines that the milk was still moving towards the refrigerator when the view was lost. All these factors lead system 100 to determine that a new item, the milk, has been added to the refrigerator.

[0081] These systems and methods can be used to maintain inventory of objects used in surgical procedures. Inventory at the end of a surgical procedure must match inventory at the start of the procedure. If the two inventory levels do not match, system 100 can issue an alert indicating that one or more objects or instruments used in the surgical procedure are missing. These systems and methods can also be used to manage inventory in refrigerators and cabinets in homes. This system and method can also be used to manage inventory in stores, vending machines, etc. Many other applications are conceivable.

[0082] The foregoing descriptions are merely illustrative and are not intended to limit the Disclosure, its application, or its use. The broad teachings of this Disclosure can be implemented in various forms. Therefore, while this Disclosure includes certain examples, the true scope of this Disclosure should not be limited in this way, for other modifications will become apparent upon consideration of the drawings, this Specification, and the following Claims. It should be understood that one or more steps in the Method can be performed in a different order (or simultaneously) without altering the principles of this Disclosure. Furthermore, while each of the embodiments is described above as having certain features, one or more of those features described in relation to any embodiment of this Disclosure can be implemented in any feature of any other embodiment, and / or combined with those features, even if such combination is not explicitly described. In other words, the embodiments described are not mutually exclusive, and the substitution of one or more embodiments with each other is within the scope of this Disclosure.

[0083] Spatial and functional relationships between multiple elements (for example, between multiple modules, multiple circuit elements, multiple semiconductor layers, etc.) are described using a variety of terms, including “connected,” “engaged,” “joined,” “adjacent,” “next to,” “above,” “below,” and “located.” Unless explicitly stated to be “direct,” a relationship between a first element and a second element, if described in the above disclosure, can be a direct relationship in which no other intervening elements are present between the first and second elements, or an indirect relationship in which one or more intervening elements (either spatially or functionally) are present between the first and second elements. When used herein, the phrase “at least one of A, B, and C” should be interpreted as meaning a logic (A or B or C) using non-exclusive OR, and not as “at least one of A, at least one of B, and at least one of C.”

[0084] In a diagram, the direction of an arrow (indicated by its tip) generally indicates the flow of information (such as data or instructions) that is relevant to the diagram. For example, if elements A and B exchange various types of information, and the information transmitted from element A to element B is relevant to the diagram, then the arrow can point from element A to element B. This one-way arrow does not mean that other information is not transmitted from element B to element A. Furthermore, with respect to information transmitted from element A to element B, element B can send a request for that information or an acknowledgment of receipt of that information to element A.

[0085] In this application, the terms “module” or “controller” may be replaced with the term “circuit,” including the definitions below. The term “module” may refer to, be part of, or include some or all of the following, such as application-specific integrated circuits (ASICs), digital, analog, or mixed analog / digital discrete circuits, digital, analog, or mixed analog / digital integrated circuits, combinational logic circuits, field-programmable gate arrays (FPGAs), processor circuits (shared, dedicated, or grouped) that execute code, memory circuits (shared, dedicated, or grouped) that store code executed by the processor circuits, other appropriate hardware components that provide the described functionality, or a system on a chip.

[0086] A module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected to a local area network (LAN), the internet, a wide area network (WAN), or a combination thereof. The functionality of any given module in this disclosure may be distributed among multiple modules connected via interface circuits. For example, multiple modules may enable load balancing. In further examples, a server module (also known as a remote or cloud module) may perform some functionality on behalf of a client module.

[0087] The term "code" as used above can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" encompasses a single processor circuit that executes some or all of the code from multiple modules. The term "group processor circuit" encompasses a processor circuit that, in combination with further processor circuits, executes some or all of the code from one or more modules. References to multiple processor circuits include multiple processor circuits on separate dies, multiple processor circuits on a single die, multiple cores in a single processor circuit, multiple threads in a single processor circuit, or a combination of the above. The term "shared memory circuit" encompasses a single memory circuit that stores some or all of the code from multiple modules. The term "group memory circuit" encompasses a memory circuit that, in combination with further memory, stores some or all of the code from one or more modules.

[0088] The term "memory circuit" is a subset of the term "computer-readable media." As used herein, the term "computer-readable media" does not include transient electrical or electromagnetic signals propagating through the media (such as on a carrier wave), and therefore, the term "computer-readable media" can be considered tangible and non-transient. Non-exclusive examples of non-transient, tangible computer-readable media include non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random-access memory circuits or dynamic random-access memory circuits), magnetic storage media (such as analog or digital magnetic tapes or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0089] The apparatus and methods described in this application can be partially or completely implemented by a dedicated computer created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The aforementioned functional blocks, flowchart components, and other elements serve as software specifications, which can be converted into computer programs through the routine work of a skilled technician or programmer.

[0090] A computer program includes processor-executable instructions stored on at least one non-temporary, tangible, computer-readable medium. A computer program may also include or depend on stored data. A computer program can encompass a basic input / output system (BIOS) that interacts with the hardware of a dedicated computer, device drivers that interact with specific devices on the dedicated computer, one or more operating systems, user applications, background services, background applications, and the like.

[0091] A computer program may include (i) descriptive text to be parsed, such as HTML (hypertext markup language), XML (extensible markup language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from the source code by a compiler; (iv) source code for execution by an interpreter; and (v) source code for compilation and execution by a just-in-time compiler. For example, source code can be written using syntax from languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language 5th revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB®, SIMULINK®, and Python®.

Claims

1. Processor and The memory that stores the instructions and A system including, where when the instruction is executed by the processor, The process involves receiving multiple images captured by the camera from an image processing system, The model is used to detect an object in one of the multiple images, Identifying the detected object using the aforementioned model and a database of previously identified objects, Tracking the movement of the identified object across a series of images from the plurality of images, Based on the aforementioned multiple images, the system detects when the identified object disappears from the camera's view, Identifying the direction of movement of the identified object, Based on the identified direction of movement, predict with higher reliability the subsequent detection of the object in the direction of movement, (i) First detection of the identified object, (ii) the last detection of the identified object, and (iii) The processor is configured to determine a result relating to the identified object based on the direction of movement of the identified object, The system wherein the last detection corresponds to the detection immediately before the identified object disappears from the camera's view.

2. The system according to claim 1, wherein determining the result includes determining that the identified object is staying in the area being observed, or that the identified object has left the area being observed.

3. The aforementioned instruction, With respect to each instance of the detection of the identified object, Assigning a timestamp to the detection of the identified object, Assigning labels to the identified objects, Assigning bounding box coordinates to the identified object, The detection history for the identified object includes storing the timestamp, the label, and the bounding box coordinates, The system according to claim 1, wherein the processor is configured to determine the first and last detections of the identified object and the direction of movement of the identified object, based on the detection history relating to the identified object.

4. The instruction, with respect to each instance of the detection of the identified object, Assigning certainty scores to the aforementioned labels, The system according to claim 3, wherein the processor is configured to increase the certainty score in conjunction with the successive detection of each of the identified objects.

5. The aforementioned instruction, The detection of the identified object in N1 images from the plurality of images, wherein N1 is an integer greater than 1. The system according to claim 1, wherein the processor is configured to determine that if the identified object disappears after the N1 images but reappears in N2 or fewer of the N1 images following the N1 images, the identified object detected in the N2 images is a continued detection of the identified object detected in the N1 images.

6. The system according to claim 5, wherein the instruction configures the processor to determine that the identified object is outside the view of the camera if the identified object is not detected in N1 + N2 of the plurality of images.

7. The plurality of images are received from the first camera and the second camera, and the command is, Using the model, detect objects in the aforementioned image, Identifying the detected object using the aforementioned model and a database of previously identified objects, In response to detecting the identified object in one of the images from the second camera, the identified object is detected in one of the images from the first camera with increased certainty. The system according to claim 1, further configuring the processor to track the movement of the identified object across the multiple images by correlating multiple detections of the identified object in the images.

8. The aforementioned instruction, To detect that the identified object is moving in the same direction across multiple images from the first and second cameras, The system according to claim 7, wherein the processor is configured to track the movement of the identified object with enhanced certainty in response to detection that the identified object is moving in the same direction across the plurality of images from the first and second cameras.

9. The system according to claim 8, wherein the processor is configured such that the instruction predicts with increased certainty the subsequent detection of the object in the direction of movement.

10. The aforementioned instruction, The detection of when the identified object disappears from the view of the first camera, The system according to claim 7, wherein the processor is configured to track the movement of the identified object in a plurality of images from the second camera in response to the identified object disappearing from the view of the first camera.

11. The aforementioned instruction, To detect when the identified object disappears from the view of the second camera, To determine the first and last detection of the identified object in the images from the first and second cameras, and to determine the direction of movement of the identified object, The system according to claim 10, wherein the processor is configured to determine a result relating to the identified object based on the first and last detection of the identified object and the direction of movement of the identified object.

12. The system according to claim 11, wherein determining the result includes determining that the identified object is staying in the area being observed, or that the identified object has left the area being observed.

13. The aforementioned instruction, With respect to each instance of the detection of the identified object, Assigning a timestamp to the detection of the identified object, Assigning labels to the identified objects, Assigning bounding box coordinates to the identified object, The detection history for the identified object includes storing the timestamp, the label, and the bounding box coordinates, The system according to claim 11, wherein the processor is configured to determine the first and last detections of the identified object and the direction of movement of the identified object by correlating the detection history relating to the identified object.

14. The aforementioned instruction, With respect to each instance of the detection of the identified object, Assigning a timestamp to the detection of the identified object, Assigning labels to the identified objects, Assigning bounding box coordinates to the identified object, The detection history for the identified object includes storing the timestamp, the label, and the bounding box coordinates, The system according to claim 7, wherein the processor is configured to track the movement of the identified object across the plurality of images by correlating the detection history relating to the identified object.

Images