Machine learning model for multi-camera multiple person tracking

Abstract

A method and system for tracking movement includes performing person detection (204) in frames from multiple video streams to identify detected images. Visual and location information from the detected images is combined (206) to generate scores for pairs of detected images across the multiple video streams and across frames of each video stream. A pairwise detection graph is generated (210) using the detected images as nodes and the scores as weighted edges. Movements of individuals are tracked (212) based on a constrained answer set programming problem determined based on matching scores and logical assumptions. Actions corresponding to the tracked movements are performed (606). Tracking patient movements in a healthcare facility can be used to inform treatment decisions by medical professionals.

Description

[Technical Field]

[0001] Related Application Information This application claims priority to U.S. Patent Application No. 63 / 424,517, filed November 11, 2022, U.S. Patent Application No. 63 / 464,247, filed May 5, 2023, and U.S. Patent Application No. 18 / 505,732, filed November 9, 2023, each of which is incorporated by reference in its entirety. This application is related to the application entitled "MULTI-CAMERA MACHINE LEARNING VIEW TRACKING," having Attorney Docket No. 22149, which is incorporated by reference in its entirety. [Background technology]

[0002] The present invention relates to video processing, and more particularly to tracking individuals across multiple cameras. 2. Description of Related Art

[0003] Large public spaces are often monitored to provide security, public safety, and healthcare services. Multiple cameras are often installed in a space to track individuals as they move through the space. However, there are areas within the space where the cameras' fields of view do not overlap, making it difficult to track individuals through such areas or when the image of the person is obstructed by an obstacle. Summary of the Invention

[0004] A method for tracking movement includes performing person detection in frames of multiple video streams to identify detected images. Visual information and location information from the detected images are combined to generate scores for pairs of detected images across the multiple video streams and across frames of each video stream. A pairwise detection graph is generated using the detected images as nodes and the scores as weighted edges. Movements of individuals are tracked based on a constrained answer set programming problem with constraints determined based on matching scores and logical assumptions. Actions are performed in response to the tracked movement.

[0005] A system for tracking motion includes a hardware processor and a memory that stores a computer program that, when executed by the hardware processor, causes the computer program to perform person detection in frames from multiple video streams to identify detected images, combine visual information and location information from the detected images to generate scores for pairs of detected images across the multiple video streams and across frames of each video stream, generate pair-wise detection graphs using the detected images as nodes and the scores as weighted edges, track motion of individuals based on a constrained answer set programming problem having constraints determined based on the matching scores and logical assumptions, and perform actions responsive to the tracked motion.

[0006] A method for tracking movements within a healthcare facility includes performing person detection in frames from multiple video streams within the healthcare facility and identifying detected images. Visual information and location information from the detected images are combined to generate scores for pairs of detected images across the multiple video streams and across frames of each video stream. A pairwise detection graph is generated using the detected images as nodes and the scores as weighted edges. Movements of individuals are tracked based on a constrained answer set programming problem with constraints determined based on matching scores and logical assumptions. A report is generated based on the tracked movements for medical professionals to make decisions regarding patient treatment.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0008] The present disclosure provides details in the following description of preferred embodiments with reference to the following figures.

[0009] [Figure 1] 1 is a diagram of an environment monitored by multiple video cameras, according to one embodiment of the present invention.

[0010] [Figure 2] 1 is a block / flow diagram of a method for tracking the movement of an individual using multiple video cameras, according to one embodiment of the present invention.

[0011] [Figure 3] 3A-3C illustrate relationships between detected person images between frames of a given video stream and between different video streams, according to one embodiment of the present invention.

[0012] [Figure 4] FIG. 10 is a block / flow diagram of tracking movement with preferred camera view according to one embodiment of the present invention.

[0013] [Figure 5] FIG. 1 is a block diagram illustrating people tracking within a healthcare management facility, according to one embodiment of the present invention.

[0014] [Figure 6] 1 is a block / flow diagram of a method for tracking and responding to the movements of an individual according to one embodiment of the present invention.

[0015] [Figure 7] FIG. 1 is a block diagram of a computing device capable of tracking and responding to the movements of an individual, according to one embodiment of the present invention.

[0016] [Figure 8] FIG. 1 illustrates an exemplary neural network architecture that may be used as part of a person detection model, according to one embodiment of the present invention.

[0017] [Figure 9] FIG. 1 illustrates an exemplary deep neural network architecture that may be used as part of a person detection model, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] Visual and location information is combined across multiple camera views to help track individuals through space. The combined information can be used to derive a scalar score between pairs of detections across frames and cameras. With a fully connected graph of all detections, representing detections as nodes and scores as edges, a constraint logic program can be used to find the best association between each detection.

[0019] Visual and location information can be combined from multiple video sources, avoiding visual bias and enabling accurate association even with occluded individuals. Furthermore, if the system makes a mistake in identifying an individual, it can correct that mistake as more information is gathered.

[0020] Referring now in detail to the figures, where like numerals represent the same or similar elements, and initially to Figure 1, an environment 100 is shown. For example, one type of environment envisioned is a mall or shopping center, which may include a common space 102 and one or more areas 104, such as stores. It should be understood that this example is provided for illustrative purposes only and should not be considered limiting.

[0021] A boundary is shown between the common space 102 and the area 104. The boundary may be any suitable physical or virtual boundary. Examples of physical boundaries include walls, ropes, or other structures that provide a physical barrier to passage from one area to another. Examples of virtual boundaries include painted lines or designations within a map of the environment 100. Virtual boundaries do not establish physical barriers to movement but may be used to identify areas within an environment. For example, a region of interest may be established next to an exhibit or display to indicate people's interest in that display. A gate 106 is shown as a passageway through the boundary where individuals are permitted to pass between the common space 102 and the area 104.

[0022] The environment 100 is monitored by a number of video cameras 114. In this embodiment, the cameras 114 are located at the gate 106, but it should be understood that such cameras can be located anywhere within the common space 102 and area 104. The video cameras 114 capture live streaming footage of individuals within the environment. A number of individuals are shown, including untracked individuals 108, represented by triangles, and tracked individuals 110, represented by circles. Also shown are tracked individuals 112, represented by squares. In some examples, all individuals are tracked individuals. In some examples, the tracked individuals 112 may be tracked to provide an interactive experience, and their movement through the environment 100 is used to trigger a response.

[0023] In addition to capturing visual information, the camera 114 may capture other types of data. For example, the camera 114 may be equipped with an infrared sensor that can read an individual's body temperature. In conjunction with visual information, this can provide the ability to remotely identify sick individuals and track their movements through the environment.

[0024] The environment 100 may include occlusions 120. For example, fixed occlusions include walls, stairs, escalators, and other barriers. Movable occlusions may include people, signs, vehicles, and other objects that may comprise a dynamic environment. The occlusions 120 may obscure individuals from a particular camera 114 from a particular angle. Thus, a person may be visible from one camera but not another, even if their visual fields overlap.

[0025] Visual and location information may be collected for each tracked individual 110 using frames from the video camera 114. Frames from each video stream may be synchronized in time so that different views of the environment 100 can be compared to each other for a given point in time.

[0026] Referring now to FIG. 2, a method for tracking individuals using multiple video streams is illustrated. Block 202 synchronizes video streams from multiple cameras 114 to identify frames that correspond to each other in time. This synchronization identifies temporal correspondences between frames from different video streams. Block 204 can extract visual information from these frames using a person detection model and an image re-identification model. The extracted visual information may include coordinates and bounding boxes for each person detected in a given frame. The two-dimensional coordinates of the frame may be projected into three-dimensional space to determine the coordinates of the person in the environment. This projection may be performed using intrinsic and extrinsic parameters of the camera. In this manner, the frames can be processed along a vision branch and a position branch to generate respective visual and position information.

[0027] The visual and location information are combined in block 206 to generate a scalar score for pairs of person images from frames. For example, two cameras viewing the same scene may each detect three people. In this example, nine scores may be generated to reflect nine different possible pairings of person images. Pairs showing the same person may have higher scores than pairs showing different people. These scores may be generated across frames from different video streams at a given time, or across frames from the same video stream at consecutive times. Exemplary scoring functions may include cosine similarity and Euclidean distance to identify the distance between visual features and location coordinates in each image pair, respectively. The distance measures may be summed to generate an output score.

[0028] Block 210 creates a pairwise person detection graph, e.g., as a fully connected graph with person detections represented as nodes and scores for pairs of person detection images represented as weighted edges between the respective nodes. Block 212 maximizes the overall score of the detection graph while maintaining the constraint of matching each person node with a unique track. Logical constraints for solving the detection graph can be set to obtain the highest possible score by matching nodes based on edge scores.

[0029] When tracking multiple individuals, constrained answer set programming may be used in block 212. Answer set programming disassociates detected people using the vision and location scores obtained from block 210. Constraints may include simple logical assumptions, such as a person in the same camera view not having the same track ID as another person, or a minimum matching score threshold to determine whether two person detections belong to the same track. These constraints may be determined based on the distribution of matching scores and logical assumptions, which may be used to find an optimal solution. The optimal solution matches people from different views and frames while satisfying the specified constraints. The output of block 212 is a set of tracks that associates all people in the detection graph.

[0030] 3, an example of pairwise matching between detected person images is shown. A first video stream 302 and a second video stream 304 are synchronized so that frames align at specific timestamps. For example, a first timestamp 310 may have a corresponding frame in the first video stream 302 and the second video stream 304, and similarly, a second timestamp 320 may have a corresponding frame in each of the video streams.

[0031] Person detection is performed on frames of a video stream. For each frame, detected person images are extracted, e.g., pixels within the bounding box of the detected person. These images are compared across video streams and timestamps, and a score is generated for each compared pair. In this way, associations can be built across multiple views of the environment and across time. For example, if a person moves behind an occlusion or out of the camera's field of view and does not appear in a particular video stream, their presence in other video streams can be used to continue tracking them through the environment.

[0032] Thus, for each pair of detected images, a visual comparison and a spatial comparison can be performed. The visual comparison may use a re-identification model to generate visual features from different camera views. The spatial comparison may identify representative coordinates in each detected image and determine the distance between these coordinates. The outputs of the visual and spatial comparisons may be combined to generate a score between "0" and "1" representing a match between the pair of images.

[0033] Visual comparison can be performed by extracting feature embeddings for person detection using re-identification models such as residual networks (ResNet) or visual transformer (ViT) networks. These models may be pre-trained on re-identification datasets to match people with similar visual features under different viewing angles and lighting conditions. Models may also be trained on multi-camera tracking datasets. Visual features between pairs of detections can be compared using any appropriate similarity metric, such as cosine similarity or Euclidean distance between feature vectors. Embeddings for the same person may have smaller distances than embeddings for different people.

[0034] For spatial or positional comparison, the person's position may be projected from the camera view to a global coordinate space. The same person in different camera views will have similar position coordinates in global space. The camera-to-global projection can be performed using the camera's intrinsic and extrinsic parameters. The spatial comparison can also be trained using a simple transformation model that can take as input the multi-view frames and the person's 2D bounding box coordinates in each view. The output of the spatial comparison may include a matching score for each person detection, learned from the projection matrix from the multi-camera tracking dataset.

[0035] Referring now to FIG. 4, a method for tracking a person across multiple camera views is shown. A person is tracked through an environment by finding a relatively high score. For example, if the score is higher than a threshold (e.g., greater than 0.9), an association may be determined between corresponding pairs of images. If no high-scoring pair exists between subsequent frames of a given video stream, but instead a high-scoring pair exists between the video streams, the view of the main camera may change. This may reflect a situation where a person moves out of the camera's field of view or behind an occlusion, but is still visible in another video stream.

[0036] Thus, block 402 generates a graph with matching scores, as described above, where nodes of the graph representing person detections may be connected to other person detections in other frames of a given video stream, and to other video streams at a given time, with edges weighted according to their combined visual and location score information.

[0037] Block 404 selects an initial camera view and also selects a person detection to track within the initial camera view. Block 406 determines whether the score between the selected person detection and a detection from a subsequent or consecutive frame of the initial camera view exceeds a threshold. If so, block 408 continues with the current camera view.

[0038] If not, block 410 determines whether the score between the selected person detection and a detection from another camera view at the same time exceeds a threshold, e.g., by looking at matching frames. If not, block 408 may continue with the current camera view because there are no matching person images in the other video streams. If the other view has an image detection score that exceeds the threshold, block 412 changes to another camera view. In this way, the selected person can be tracked as they move from one camera view to another or pass behind an occlusion. The view change may be implemented in a user interface that receives multiple video streams and automatically selects a video stream to display to track a given individual.

[0039] 5, an illustration of activity recognition in the context of a healthcare facility 500 is shown. As a patient moves through the healthcare facility 500, their activity 506 may be monitored by video cameras installed in the facility 500. Person tracking 508 may be used to monitor the patient's activity 506, thereby enabling an understanding of the patient's behavior and healthcare needs. For example, person tracking 508 may provide information about how the patient interacts with the treatment system 504.

[0040] The health care facility may include one or more medical professionals 502 that provide information related to patient behavior and information provided by a treatment system 504. When person tracking 508 is performed using patient behavior 506, information about patient behavior may be automatically generated. Behaviors that reflect unmet needs or risks may be included in a report for the medical professional 502 to use in decision-making. For example, based on the report, the medical professional 502 may modify the patient's treatment to promote healthy behaviors.

[0041] The different elements of the healthcare facility 500 may communicate with each other via a network 510, for example, using any suitable wired or wireless communication protocol and medium, thereby enabling a medical professional 502 to receive information related to patient activity 506 from people tracking 508.

[0042] In other embodiments, tracking can be performed in heavily obstructed or complex environments with multiple cameras, such as museums, shopping malls, airports, etc. Tracking of individuals at crowded events, such as sporting events, can be achieved using cameras mounted on unmanned aerial vehicles. Tracking can also be used to backtrace the path taken by an individual for offline analysis or other non-real-time investigations.

[0043] 6, a method for tracking and responding to an individual's movements is shown. Block 602 identifies an individual in a video stream. The individual is selected in a user interface, and block 604 tracks the individual's movements across the camera view.

[0044] Based on the tracked movements, block 606 executes a response action. As described above, the response action may include a healthcare-oriented action, such as generating a report used in medical decision-making. In some cases, the response action may include a security action, such as locking or unlocking a door, granting or denying access, or alerting security personnel. The response action may be executed automatically upon detecting that the tracked movements meet appropriate criteria. For example, if an individual is tracked in a location they are not authorized to access, a security action may be automatically triggered. If the individual's movements indicate a negative health event, a healthcare response may be automatically triggered.

[0045] 7, an exemplary computing device 700 according to an embodiment of the present invention is shown, which is configured to track motion across multiple cameras.

[0046] Computing device 700 may be embodied as any type of computational or computing device capable of performing the functions described herein, such as, but not limited to, a computer, a server, a rack-based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and / or a user electronic device. Additionally or alternatively, computing device 700 may be embodied as one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically separated computing device.

[0047] 7, computing device 700 illustratively includes a processor 710, an input / output subsystem 720, a memory 730, a data storage device 740, and a communications subsystem 750, and / or other components and devices typically found in a server or similar computing device. In other embodiments, computing device 700 may include other or additional components (e.g., various input / output devices) such as those typically found in a server computer. Furthermore, in some embodiments, one or more of the illustrative components may be incorporated into or otherwise form part of another component. For example, memory 730, or portions thereof, may be incorporated into processor 710 in some embodiments.

[0048] The processor 710 may be embodied as any type of processor capable of performing the functions described herein, and may be embodied as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.

[0049] Memory 730 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. During operation, memory 730 may store various data and software used during operation of computing device 700, such as an operating system, applications, programs, libraries, and drivers. Memory 730 is communicatively coupled to processor 710 via I / O subsystem 720 and may be embodied as circuits and / or components for facilitating input / output operations with processor 710, memory 730, and other components of computing device 700. For example, I / O subsystem 720 may be embodied as or otherwise include a memory controller hub, an input / output control hub, a platform controller hub, integrated control circuitry, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems for facilitating input / output operations. In some embodiments, I / O subsystem 720 may form part of a system-on-chip (SOC) and be integrated with processor 710, memory 730, and other components of computing device 700 on a single integrated circuit chip.

[0050] The data storage device 740 may be embodied as any type of device or device configured for short-term or long-term storage of data, such as, for example, a memory device and circuitry, a memory card, a hard disk drive, a solid-state drive, or other data storage device. The data storage device 740 may store program code 740A for person detection, program code 740B for tracking movement, and / or program code 740C for generating reports regarding the tracked movement. The communications subsystem 750 of the computing device 700 may be embodied as any network interface controller or other communications circuitry, device, or collection thereof that can enable communications between the computing device 700 and other remote devices over a network. The communications subsystem 750 may be configured to achieve such communications using any one or more communications technologies (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand, Bluetooth, Wi-Fi, WiMAX, etc.).

[0051] As shown, computing device 700 may also include one or more peripheral devices 760. Peripheral devices 760 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 760 may include a display, a touch screen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0052] Of course, computing device 700 may include other elements (not shown) or omit certain elements, as would readily occur to one skilled in the art. For example, various other sensors, input devices, and / or output devices may be included in computing device 700, depending on the particular implementation of the same, as would be readily understood by one skilled in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memory, etc. may be utilized in a variety of configurations. These and other variations of processing system 700 will be readily contemplated by one skilled in the art in view of the teachings of the present invention provided herein.

[0053] 8 and 9, exemplary neural network architectures are shown that can be used to implement portions of the present model, such as person detection 802. Neural networks are generalized systems whose capabilities and accuracy improve with exposure to additional empirical data. Neural networks learn by exposure to empirical data. During training, the neural network stores and adjusts weights that are applied to input empirical data. Applying the adjusted weights to the data can identify that data belongs to a specific predefined class from a set of classes and output the probability that the input data belongs to each class.

[0054] Empirical data (also called training data) from a set of examples is formatted as a string of values ​​and fed to the input of a neural network. Each example is associated with a known outcome or output. Each column is represented as an (x,y) pair, where x represents the input data and y represents the known output. The input data can be of various data types and contain multiple different values. The network can have one input node for each value that makes up the example input data, and a separate weight can be applied to each input value. The input data can be formatted, for example, as a vector, array, or string, depending on the architecture of the neural network being built and trained.

[0055] Neural networks "learn" by comparing neural network outputs generated from input data with known values ​​from examples and adjusting stored weights to minimize the difference between the output value and the known value. Adjustments can be made to the stored weights through backpropagation, where the weight's influence on the output value is determined by calculating a mathematical gradient and adjusting the weight in a way that shifts the output toward the minimum difference. This optimization, called gradient descent, is a non-limiting example of how training occurs. A subset of examples with known values ​​that were not used in training can be used to test and validate the accuracy of the neural network.

[0056] During operation, a trained neural network can be used on new data not previously used for training or validation through generalization. The adjusted neural network weights can be applied to the new data, and the weights estimate the function developed from the training examples. The parameters of the estimated function, captured by the weights, are based on statistical inference.

[0057] In a layered neural network, nodes are arranged in layers. An exemplary simple neural network has an input layer 820 of source nodes 822 and a single computational layer 830 with one or more computational nodes 832 that also function as output nodes, with a single computational node 832 for each possible category into which an input example may be classified. The input layer 820 may have a number of source nodes 822 equal to the number of data values ​​812 in the input data 810. The data values ​​812 in the input data 810 may be represented as a column vector. Each computational node 832 in the computational layer 830 generates a weighted linear combination of values ​​from the input data 810 provided to the input node 820 and applies a differentiable nonlinear activation function to the sum. An exemplary simple neural network can perform classification on linearly separable examples (e.g., patterns).

[0058] A deep neural network, such as a multilayer perceptron, may have an input layer 820 of source nodes 822, one or more computational layers 830 with one or more computational nodes 832, and an output layer 840 with one output node 842 for each possible category into which an input example may fall. The input layer 820 may have a number of source nodes 822 equal to the number of data values ​​812 in the input data 810. The computational nodes 832 in the computational layer 830 are also referred to as hidden layers because they are located between the source nodes 822 and the output nodes 842 and are not directly observed. Each node 832, 842 in the computational layer generates a weighted linear combination of values ​​from the values ​​output by the nodes in the previous layer and applies a nonlinear activation function that is differentiable over the range of the linear combination. The weights applied to the values ​​from each previous node may be, for example, w1, w2, ... w n-i ,w n The output layer provides the network's overall response to the input data. Deep neural networks can be fully connected, where each node in a computational layer is connected to every node in the previous layer, or the connections between layers can have other configurations. If links between nodes are missing, the network is said to be partially connected.

[0059] Training a deep neural network involves two phases: a forward phase, in which the weights of each node are fixed and inputs are propagated through the network, and a backward phase, in which error values ​​are propagated back through the network and the weight values ​​are updated.

[0060] Computational nodes 832 in one or more computational (hidden) layers 830 perform nonlinear transformations on the input data 812 to generate a feature space. Classes or categories may be more easily separated in the feature space than in the original data space.

[0061] The embodiments described herein may be entirely hardware, entirely software, or contain both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0062] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer-readable medium may include any apparatus that stores, communicates, propagates, or transports a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or propagation medium. The medium may include computer-readable storage media such as semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks.

[0063] Each computer program can be tangibly stored on a machine-readable storage medium or device (e.g., program memory or magnetic disk) readable by a general-purpose or special-purpose programmable computer to configure and control the operation of the computer when the storage medium or device is read by the computer to perform the procedures described herein. The system of the present invention can also be considered to be embodied in a computer-readable storage medium configured with a computer program, where the configured storage medium causes the computer to operate in a particular, predetermined manner to perform the functions described herein.

[0064] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory employed during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0065] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0066] As employed herein, the terms “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.) that may be on-board or off-board or dedicated for use by the hardware processor subsystem.

[0067] In some embodiments, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code for achieving a specified result.

[0068] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0069] These and other variations of the hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0070] References herein to "one embodiment" or "one embodiment" of the present invention, as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "in one embodiment" or "in one embodiment," as well as any other variations thereof, in various places throughout this specification do not necessarily all refer to the same embodiment. However, it should be understood that features of one or more embodiments may be combined given the teachings of the present invention provided herein.

[0071] For example, in the case of "A / B," the use of any of the following " / ," "and / or," "at least one," such as "A and / or B" or "at least one of A and B" will be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), the selection of only the first and third listed alternatives (A and C), the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as there are listed items.

[0072] The foregoing is understood in all respects to be illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but from the claims which are interpreted in accordance with the full breadth permitted by the patent laws. It will be understood that the embodiments shown and described herein are merely exemplary of the invention, and that those skilled in the art could make various modifications without departing from the scope and spirit of the invention. Various other feature combinations could be implemented by those skilled in the art without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is set forth in the appended claims.

Claims

1. 1. A method for tracking movement, comprising: performing 204 person detection in frames from the plurality of video streams to identify detected images; combining (206) visual and positional information from the detected images to generate scores for pairs of detected images across the plurality of video streams and across frames of each video stream; generating 210 a pairwise detection graph using the detected images as nodes and the scores as weighted edges; Tracking 212 the movements of the individual based on a constrained answer set programming problem with constraints determined based on the matching scores and logical assumptions; and performing (606) an action responsive to the tracked movement.

2. 10. The method of claim 1, The method further comprising synchronizing the multiple video streams to determine a temporal correspondence between frames of the multiple video streams.

3. 10. The method of claim 1, The method further comprising extracting the visual information based on visual similarity between the detected images.

4. 10. The method of claim 1, The method further comprising: extracting the position information based on a projection of two-dimensional coordinates onto a three-dimensional environment for the detected image; and determining a distance between the projected coordinates.

5. 10. The method of claim 1, The method, wherein generating the pairwise detection graph includes determining edges between detected images from different frames of the same video stream and determining edges between detected images from different video streams at corresponding times.

6. 10. The method of claim 1, The method, wherein the actions include generating a report based on the tracked patient movements to enable a medical professional to make decisions related to the patient's care.

7. 10. The method of claim 1, The method, wherein tracking the individual's movements relates to movements within a healthcare facility and the plurality of video streams are generated by video cameras within the healthcare facility.

8. 10. The method of claim 1, The method, wherein combining the visual information and the position information includes adding an output from a visual branch to an output from a position branch.

9. 9. The method of claim 8, The method, wherein the vision branch processes the detected image with a re-identification model.

10. 1. A system for tracking movement, comprising: a hardware processor (710); and a memory (740) for storing a computer program, the computer program, when executed by a hardware processor, causing the hardware processor to: performing 204 person detection in frames from the plurality of video streams to identify detected images; combining (206) visual and positional information from the detected images to generate scores for pairs of detected images across the plurality of video streams and across frames of each video stream; generating 210 a pairwise detection graph using the detected images as nodes and the scores as weighted edges; Tracking 212 the movements of the individual based on a constrained answer set programming problem with constraints determined based on the matching scores and logical assumptions; performing (606) an action responsive to the tracked movement; and a system causing the action to be performed.

11. 11. The system of claim 10, The computer program further causes the hardware processor to synchronize the multiple video streams and determine temporal correspondences between frames of the multiple video streams.

12. 11. The system of claim 10, The computer program further causes the hardware processor to extract the visual information based on visual similarity between detected images.

13. 11. The system of claim 10, The computer program further causes the hardware processor to extract the position information based on projection of two-dimensional coordinates onto a three-dimensional environment for the detected image and determine a distance between the projected coordinates.

14. 11. The system of claim 10, The computer program further causes the hardware processor to determine edges between detected images from different frames of the same video stream and determine edges between detected images from different video streams at corresponding times.

15. 11. The system of claim 10, The system, wherein the operations include generating a report based on the tracked patient movements to enable a medical professional to make decisions related to the patient's treatment.

16. 11. The system of claim 10, The system wherein the tracked movements of the individual relate to movements within a healthcare facility, and the plurality of video streams are generated by video cameras within the healthcare facility.

17. 11. The system of claim 10, The computer program further causes the hardware processor to combine the visual information and the position information by adding an output from a visual branch to an output from a position branch.

18. 18. The system of claim 17, The system wherein the visual branch includes a re-identification model for processing the detected image.

19. 1. A method for tracking movement within a healthcare facility, comprising: performing person detection (204) in frames from a plurality of video streams within a healthcare facility to identify detected images; combining (206) visual and positional information from the detected images to generate scores for pairs of detected images across the plurality of video streams and across frames of each video stream; generating 210 a pairwise detection graph using the detected images as nodes and the scores as weighted edges; Tracking 212 the movements of the individual based on a constrained answer set programming problem with constraints determined based on the matching scores and logical assumptions; and generating (606) a report based on the tracked movements for a medical professional to make decisions related to the patient's treatment.

20. 20. The method of claim 19, The method, wherein generating the pairwise detection graph includes determining edges between detected images from different frames of the same video stream and determining edges between detected images from different video streams at corresponding times.

Images