Machine learning model for multi-camera multi-person tracking

Abstract

Methods and systems for tracking motion include performing (204) person detection on individual frames from multiple video streams to identify detection frames. Visual and location-based information from the detection frames is combined (206) to generate scores for pairs of detection frames across the multiple video streams and across individual frames of the respective video streams. A pairwise detection graph is constructed (210) using the detection frames as nodes and the scores as weighted edges. The motion of an individual is tracked (212) based on a constrained response set programming problem, where the constraints are determined based on matching scores and logical assumptions. An action responsive to the tracked motion is performed (606).Tracking a patient's movement within a healthcare facility can inform treatment decisions made by medical staff.

Description

REGISTRATION INFORMATION

[0001] This application claims priority to U.S. Patent Application No. 63 / 424,517, filed on November 11, 2022, U.S. Patent Application No. 63 / 464,247, filed on May 5, 2023, and U.S. Patent Application No. 18 / 505,732, filed on November 9, 2023, each of which is incorporated herein by reference in its entirety. This application is related to an application entitled "MULTI-CAMERA MACHINE LEARNING VIEW TRACKING," having Attorney Docket No. 22149, which application is incorporated herein by reference in its entirety. BACKGROUNDTechnical field

[0002] The present invention relates to video processing and, in particular, to tracking people across multiple cameras. Description of the state of the art

[0003] Large public spaces can be monitored to ensure security, public safety, and healthcare. Multiple cameras can be installed in a given space to track people as they move through it. However, there may be some locations in the space where the camera views do not overlap, making it difficult to track a person in those locations or in situations where occlusions block an image of the person. SUMMARY

[0004] A method for motion tracking includes performing person detection on individual frames from multiple video streams to identify detection frames. Visual and location-based information from the detection frames is combined to generate scores for pairs of detection frames across the multiple video streams and across individual frames of the respective video streams. A pairwise detection graph is constructed using the detection frames as nodes and the scores as weighted edges. The movement of an individual is tracked based on a constrained-answer-set programming problem, where the constraints are determined based on matching scores and logical assumptions. An action responsive to the tracked movement is performed.

[0005] A motion tracking system includes a hardware processor and memory that stores a computer program.When executed by the hardware processor, the computer program causes the hardware processor to perform person detection in frames from multiple video streams to identify detection images, combine visual and location information from the detection images to generate scores for pairs of detection images across the multiple video streams and across frames of the respective video streams, generate a pairwise detection graph using the detection images as nodes and the scores as weighted edges, track the movement of a person based on a constrained response set programming problem, where the constraints are determined based on matching scores and logical assumptions, and perform an action in response to the tracked movement.

[0006] A method for tracking movement in a healthcare facility includes performing person detection on individual frames from multiple video streams in a healthcare facility to identify detection images. Visual and location-based information from the detection images is combined to generate scores for pairs of detection images across the multiple video streams and across individual frames of the respective video streams. A pairwise detection graph is constructed using the detection images as nodes and the scores as weighted edges. The movement of a person is tracked based on a constrained-answer-set programming problem, where the constraints are set based on matching scores and logical assumptions.Based on the tracked movements, a report is prepared for the medical professional to help decide on the patient's treatment.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The disclosure is presented in detail in the following description of preferred embodiments with reference to the following figures, wherein: Fig. 1 is a diagram of an environment monitored by multiple video cameras according to an embodiment of the present invention; Fig. 2 is a block / flow diagram of a method for tracking the movement of a person using multiple video cameras, according to an embodiment of the present invention; Fig. 3 is a diagram illustrating the relationships between detected person images between frames of a given video stream and between different video streams according to an embodiment of the present invention; Fig. 4 is a block / flow diagram of motion tracking with a preferred camera view according to an embodiment of the present invention; Fig. 5 is a block diagram for tracking individuals in a healthcare facility according to an embodiment of the present invention; Fig. 6 is a block / flow diagram of a method for tracking and responding to the movements of a person according to an embodiment of the present invention; Fig. 7 is a block diagram of a computing device that can track and respond to a person's movements, according to an embodiment of the present invention; Fig. 8 is a diagram illustrating an exemplary architecture of a neural network that may be used as part of a person recognition model according to an embodiment of the present invention; and Fig. 9 is a diagram illustrating an example deep neural network architecture that may be used as part of a person recognition model according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Visual and positional information can be combined across multiple camera views to support tracking people through a space. The combined information can be used to obtain a scalar score between pairs of detections across frames and cameras. Using a fully connected graph of all detections, with detections represented as nodes and scores as weighted edges, a constraint logic program can be used to find the best connections between each detection.

[0010] Visual information and positioning data can be combined from multiple video sources, eliminating visual distortions and enabling accurate identification even of occluded individuals. Should the system make an error in identifying a person, it can be corrected as more information is collected.

[0011] In the figures where the same numbers represent the same or similar elements, and first in Fig. 1, an environment 100 is depicted. An example of such an environment is a shopping mall, which may include a common space 102 and one or more areas 104, such as a store. This example is for illustrative purposes only and should not be considered limiting.

[0012] A boundary is depicted between the common space 102 and the area 104. The boundary can be any suitable physical or virtual boundary. Examples of physical boundaries include walls and ropes—anything that provides a physical barrier to movement from one area to another. Examples of virtual boundaries include a painted line and a label on a map of the environment 100. Virtual boundaries do not provide a physical barrier to movement, but can still be used to identify regions within the environment. For example, an area of ​​interest can be designated next to an exhibit or display to indicate visitor interest in that display. A gate 106 is depicted as a passage through the boundary, allowing people to pass between the common space 102 and the area 104.

[0013] The environment 100 is monitored by an array of video cameras 114. Although in this illustration, the cameras 114 are positioned at the gate 106, these cameras may be placed anywhere in the common room 102 and area 104. The video cameras 114 capture live video images of people in the environment. A variety of people are shown, including untracked people 108, represented as triangles, and tracked people 110, represented as circles. Also shown is a tracked person of interest 112, represented as a square. In some examples, all of the people may be tracked people. In some examples, the tracked person of interest 112 may be tracked to provide an interactive experience, using their movement through the environment 100 to trigger responses.

[0014] In addition to capturing visual information, the cameras 114 can also capture other types of data. For example, the cameras 114 can be equipped with infrared sensors that can measure a person's body temperature. Combined with the visual information, this can provide the ability to remotely identify sick individuals and track their movement through the environment.

[0015] The environment 100 may include occlusions 120. Fixed obstacles may include, for example, walls, stairs, escalators, and other barriers. Moving obstacles may include people, signs, vehicles, and other objects that may form a dynamic environment. The occlusions 120 may prevent people from being visible from certain angles and to certain cameras 114. Thus, a person visible from one camera may not be visible to another camera, even if their fields of view otherwise overlap.

[0016] Visual and location-based information can be collected for each tracked person 110 using individual images from the video cameras 114. The images from the respective video streams can be synchronized in time so that different views of the environment 100 can be compared at specific times.

[0017] Fig. shows a method for tracking people using multiple video streams. In block 202, the video streams from multiple cameras 114 are synchronized so that temporally coincident images can be identified. This synchronization identifies temporal correspondences between the individual images of different video streams. Visual information can be extracted from these individual images in block 204 using a person detection model and a frame re-identification model. The extracted visual information can include coordinates and a bounding box for each person detected in a particular image. The two-dimensional coordinates of the image can be projected into a three-dimensional space to determine coordinates for the person in the environment. This projection can be performed using intrinsic and extrinsic parameters of the camera.Thus, the images can be processed along a visual branch and a location branch to generate corresponding visual and location information.

[0018] The visual and location-related information may be combined in block 206, and a scalar score may be created for pairs of person images from the individual images. For example, two cameras viewing the same scene may each detect three people. In this example, nine scores may be generated to reflect the nine different possible pairings of the person images. Pairs showing the same person may have a higher score than pairs showing different people. These scores may be created for images from different video streams at a given time and also for images from the same video stream at consecutive times. Example scoring functions may be cosin similarity and Euclidean distance to determine the distance between visual features and location coordinates, respectively, in the respective image pairs.The distance measures can be added together to produce the initial score.

[0019] Block 210 creates a pairwise person detection graph, e.g., in the form of a fully connected graph, that represents person detections as nodes and that represents the scores for a pair of person detection images as a weighted edge between the respective nodes. Block 212 maximizes the overall score of the detection graph while maintaining constraints to connect each person node to a unique track. Logical constraints on the solution to the detection graph can be set for the highest possible score by matching the nodes based on their edge values.

[0020] When tracking multiple people, constrained response set programming may be used in block 212. Response set programming resolves the association of detected people based on the visual and location scores obtained in block 210. The constraints may include simple logical assumptions, such as that a person in the same camera view does not have the same lane ID as another person, and minimum matching thresholds to determine whether two person detections belong to the same lane. These constraints may be determined based on the distribution of match scores and logical assumptions, which can be used to find an optimal solution. The optimal solution matches people across different views and frames while satisfying the specified constraints. The output of block 212 may then be the set of lanes that associate all people in the detection graph.

[0021] In Fig. An example of pairwise matching between detected person images is shown. A first video stream 302 and a second video stream 304 are synchronized so that the images are aligned to specific timestamps. For example, a first timestamp 310 may have a corresponding frame in the first video stream 302 and the second video stream 304, and similarly, a second timestamp 320 may have a corresponding frame in each of the video streams.

[0022] Person detection is performed using the frames from the video streams. For each individual frame, images of the detected person can be extracted, for example, with pixels within the detected person's bounding box. These images can be compared across video streams and timestamps, and a score can be created for each compared pair. In this way, associations can be made across multiple views of the environment and over time. If a particular person is not visible in a particular video stream—for example, because they have disappeared behind an occlusion or left the camera's field of view—their presence in other video streams can be used to continue tracking them through the environment.

[0023] For each pair of detected images, a visual comparison and a spatial comparison can be performed. The visual comparison can use a re-identification model to generate visual features from different camera views. During the spatial comparison, a representative coordinate can be determined for each captured image and the distance between these coordinates can be determined. The results of the visual comparison and the spatial comparison can be combined to determine a score between 0 and 1, indicating a match between the images in the pair.

[0024] Visual comparison can be performed by extracting feature embeddings for person detection using a re-identification model, such as a residual network (ResNet) or a vision transformer network (ViT). These models can be trained on re-recognition datasets to detect a person with similar visual features under different view angles and lighting conditions. The model can also be trained on a multi-camera tracking dataset. The visual features between a pair of detections can be compared using any suitable similarity metric, such as cosine similarity or Euclidean distance between feature vectors. The embeddings for the same person may have a closer distance than the embeddings for different people.

[0025] For spatial or location-based comparisons, the person's position can be projected from the camera views to a global coordinate space. The same person in different camera views will have similar position coordinates in global space. The projection from the camera to global space can be performed using intrinsic and extrinsic camera parameters. Spatial comparison can also be learned using a simple transformation model that can take multiple views and two-dimensional bounding box coordinates of people in each view as input. The output of the spatial comparison can contain agreement scores for each person detection learned from the projection matrix from the multi-camera tracking datasets.

[0026] Fig. shows a method for tracking a person across multiple camera views. A person can be tracked through the environment by finding relatively high scores. For example, if a score is above a threshold (e.g., above 0.9), an association can be determined between the corresponding image pair. If there is no high score pair between consecutive frames of a particular video stream, but instead a high score pair between video streams, a main camera view can be changed. This can reflect a situation where a person has left the field of view of one camera or moved behind an occlusion but remains visible in another video stream.

[0027] In block 402, a graph with match scores is created as described above. The nodes of the graph representing person detections can be connected to other person detections in other frames of a particular video stream and to other video streams at a particular time, with the edges weighted according to their combined visual and location-based scoring information.

[0028] Block 404 selects an initial camera perspective and further selects a person detection within the initial camera perspective for tracking. Block 406 determines whether the score between the selected person detection and a detection from a subsequent or consecutive image of the original camera view is above the threshold. If so, block 408 continues with the current camera view.

[0029] If not, block 410 determines whether a match between the selected person detection and a detection from another camera view at the same time is above the threshold, e.g., in the case of matching images. If not, block 408 may continue with the current camera view because there is no matching person image in other video streams. If another view has an image detection score above the threshold, block 412 switches to that other camera view. In this way, the selected person can be tracked as they move from one camera view to another and as they pass behind occlusions. Switching views can be implemented in a user interface that receives multiple video streams and automatically selects a video stream to display for tracking a specific person.

[0030] Fig. shows a diagram of action detection in the context of a healthcare facility 500. As a patient moves through a healthcare facility 500, their activity 506 can be monitored by video cameras installed in the facility 500. Person tracking 508 can be used to monitor patient activity 506, which can help understand patient behavior and healthcare needs. For example, person tracking 508 can provide information about how the patient interacts with treatment systems 504.

[0031] The healthcare facility may include one or more healthcare professionals 502 who provide information about the patient's activity and the information provided by the treatment systems 504. Because the person tracking 508 is performed based on the patient activity 506, information about the patient's behavior may be automatically generated. Behaviors that reflect an unmet need or a risk may be included in a report that the healthcare professional 502 can use in decision-making. Based on the report, the healthcare professional 502 may, for example, adjust the patient's treatment to promote healthy activity.

[0032] The various elements of healthcare facility 500 can communicate with each other via a network 510, e.g., using a suitable wired or wireless communication protocol and medium. Healthcare professional 502 can obtain information about the activity of patient 506 from personal tracking 508.

[0033] In other embodiments, tracking can be performed in any highly obscured or complex environment using multiple cameras, such as museums, shopping malls, and airports. Tracking people at crowded events, such as sporting events, can be performed using cameras on unmanned aerial vehicles. Tracking can also be used to retrace a person's path for offline analysis and other non-real-time investigations.

[0034] Fig. shows a method for tracking and responding to a person's movements. In block 602, a person is identified in a video stream. This person can be selected in a user interface, and the person's movement can be tracked across camera views in block 604.

[0035] Based on the tracked movement, a response action is performed in block 606. As described above, the response action may include a healthcare-focused action, such as generating a report that can be used for medical decision-making. In some cases, the response action may be a security action, such as locking or unlocking a door, allowing or denying access, or summoning security personnel. The response may occur automatically once it is determined that the tracked movement meets an appropriate criterion. For example, if the individual is located in a location to which they are not authorized, a security action may be automatically triggered. If the individual's movements indicate an adverse health event, a healthcare action may be automatically triggered.

[0036] In Fig. Figure 7 illustrates an exemplary computing device 700 according to an embodiment of the present invention. Computing device 700 is configured to track motion via multiple cameras.

[0037] Computing device 700 may be embodied as any type of computing or computer device capable of performing the functions described herein, including, without limitation, a computer, a server, a rack-based server, a blade server, a workstation, a desktop computer, a laptop, a notebook, a tablet computer, a mobile computing device, a portable computing device, a network, a web device, a distributed computing system, a processor-based system, and / or a consumer electronic device. Additionally or alternatively, computing device 700 may be embodied as one or more compute sleds, storage sleds, or other racks, sleds, compute enclosures, or other components of a physically disaggregated computing device.

[0038] As in Fig. 7, the computing device 700 includes the processor 710, an input / output subsystem 720, a memory 730, a data storage device 740, and a communications subsystem 750, and / or other components and devices typically found in a server or similar computing device. In other embodiments, the computing device 700 may include other or additional components typically found in a server computer (e.g., various input / output devices). Furthermore, in some embodiments, one or more of the illustrated components may be integrated with or otherwise form part of another component. For example, in some embodiments, the memory 730 or portions thereof may be integrated with the processor 710.

[0039] Processor 710 may be embodied as any type of processor capable of performing the functions described herein. Processor 710 may be embodied as a single processor, multiple processors, central processing units (CPU(s)), graphics processing units (GPU(s)), single- or multi-core processor(s), digital signal processor(s), a microcontroller, or other processor(s) or processing / control circuit(s).

[0040] Memory 730 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, memory 730 may store various data and software used during operation of computing device 700, such as operating systems, applications, programs, libraries, and drivers. Memory 730 is communicatively coupled to processor 710 via I / O subsystem 720, which may be embodied as circuits and / or components to facilitate input / output operations with processor 710, memory 730, and other components of computing device 700. For example, I / O subsystem 720 may be embodied as memory controller hubs, input / output control hubs, platform controller hubs, integrated control circuits, firmware devices, communication links (e.g., point-to-point links, bus links), wires, cables, optical fibers, traces, etc.) and / or other components and subsystems to facilitate input / output operations. In some embodiments, the I / O subsystem 720 may be part of a system-on-chip (SOC) and may be packaged on a single integrated circuit chip along with the processor 710, the memory 730, and other components of the computing device 700.

[0041] The data storage device 740 may be embodied as any type of device or devices configured for the short- or long-term storage of data, such as memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. The data storage device 740 may store program code 740A for person detection, 740B for tracking movement, and / or 740C for generating a report on the tracked movement. Any or all of these program code blocks may be included in a particular computer system. The communications subsystem 750 of the computing device 700 may be embodied as any network interface controller or other communications circuitry, device, or collection thereof capable of enabling communication between the computing device 700 and other remote devices over a network.The communication subsystem 750 may be configured to use one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, etc.) to perform such communication.

[0042] As illustrated, computing device 700 may also include one or more peripherals 760. Peripherals 760 may include any number of additional input / output devices, interface devices, and / or other peripherals. In some embodiments, peripherals 760 may include, for example, a display, a touchscreen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripherals.

[0043] Of course, computing device 700 may also include other elements (not shown) as would be readily contemplated by one of ordinary skill in the art, or omit certain elements. For example, various other sensors, input devices, and / or output devices may be included in computing device 700, depending on the particular implementation thereof, as would be readily understood by one of ordinary skill in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memory, etc., may be used in various configurations. These and other variations of processing system 700 may be readily contemplated by one of ordinary skill in the art in light of the teachings of the present invention contained herein.

[0044] With reference to the Fig. 8 and Fig.Figure 9 illustrates exemplary neural network architectures that can be used to implement portions of the present models, such as person detection 802. A neural network is a generalized system that improves its performance and accuracy through exposure to additional empirical data. The neural network is trained through exposure to the empirical data. During training, the neural network stores and adjusts a plurality of weights applied to the incoming empirical data. By applying the adjusted weights to the data, the data can be identified as belonging to a particular predefined class from a set of classes, or a probability that the input data belongs to each of the classes can be output.

[0045] The empirical data, also called training data, from a series of examples can be formatted as a sequence of values ​​and fed into the input of the neural network. Each example can be associated with a known result or output. Each example can be represented as a pair (x, y), where x represents the input data and y represents the known output. The input data can include a variety of different data types and several distinct values. The network can have an input node for each value that makes up the example's input data, and each input value can have its own weighting applied. The input data can be formatted, for example, as a vector, array, or string, depending on the architecture of the neural network to be constructed and trained.

[0046] The neural network "learns" by comparing the output generated from the input data with the known values ​​of the examples and adjusting the stored weights to minimize the differences between the output values ​​and the known values. Adjusting the stored weights can be done by backpropagation, where the effect of the weights on the output values ​​can be determined by calculating the mathematical gradient and adjusting the weights in a way that shifts the output toward a minimum difference. This optimization, known as gradient descent, is a non-limiting example of how training can be performed. A subset of examples with known values ​​that were not used for training can be used to test and validate the accuracy of the neural network.

[0047] During operation, the trained neural network can be applied to new data that was not previously used for training or validation through generalization. The adjusted weights of the neural network can be applied to the new data, with the weights estimating a function developed from the training examples. The parameters of the estimated function, captured by the weights, are based on statistical inference.

[0048] In layered neural networks, the nodes are arranged in layers. An exemplary simple neural network has an input layer 820 consisting of source nodes 822 and a single computation layer 830 with one or more computation nodes 832 that also function as output nodes, with a single computation node 832 for each possible category into which the input example could be classified. An input layer 820 may have a number of source nodes 822 equal to the number of data values ​​812 in the input data 810. The data values ​​812 in the input data 810 may be represented as a column vector. Each computation node 832 in the computation layer 830 generates a linear combination of weighted values ​​from the input data 810 fed to the input nodes 820 and applies a nonlinear activation function that is differentiable to the sum.The exemplary simple neural network can perform classification of linearly separable examples (e.g., patterns).

[0049] A deep neural network, such as a multi-layer perceptron, may have an input layer 820 with source nodes 822, one or more computation layers 830 with one or more computation nodes 832, and an output layer 840 in which there is a single output node 842 for each possible category into which the input example could be classified. An input layer 820 may have a number of source nodes 822 equal to the number of data values ​​812 in the input data 810. The computation nodes 832 in the computation layer(s) 830 may also be referred to as hidden layers because they lie between the source node(s) 822 and the output node(s) 842 and are not directly observed.Each node 832, 842 in a computation layer generates a linear combination of weighted values ​​from the values ​​output by the nodes in a previous layer and applies a nonlinear activation function that is differentiable over the range of the linear combination. The weights applied to the value of each previous node can be denoted, for example, by w1, w2, ... w. (n-1) (,) w(n) The output layer provides the overall response of the network to the input data. A deep neural network can be fully connected, meaning every node in a computational layer is connected to all other nodes in the previous layer, or it can have other configurations of connections between layers. If there are no connections between nodes, the network is said to be partially connected.

[0050] Training a deep neural network can involve two phases: a forward phase in which the weights of each node are fixed and the input is propagated through the network, and a backward phase in which an error value is propagated backward through the network and the weight values ​​are updated.

[0051] The computation nodes 832 in the one or more (hidden) computation layers 830 perform a nonlinear transformation of the input data 812, generating a feature space. Classes or categories can be more easily distinguished in the feature space than in the original data space.

[0052] The embodiments described herein may be entirely hardware, entirely software, or may include both hardware and software elements. In a preferred embodiment, the present invention is implemented in software, including, but not limited to, firmware, resident software, microcode, etc.

[0053] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or instruction execution system. A computer-usable or computer-readable medium may include any device that stores, communicates, propagates, or transports the program for use by or in connection with the instruction execution system, device, or apparatus. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device), or a propagation medium. The medium may be a computer-readable storage medium, such asa semiconductor or solid-state memory, a magnetic tape, a removable computer disk, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, and an optical disk, etc.

[0054] Each computer program may be stored in a machine-readable storage medium or device (e.g., program memory or magnetic disk) that can be read by a programmable general-purpose or special-purpose computer to configure and control the operation of a computer when the storage medium or device is read by the computer to perform the methods described herein. The inventive system may also be considered to be embodied in a computer-readable storage medium configured with a computer program, the storage medium so configured causing a computer to operate in a specific and predefined manner to perform the functions described herein.

[0055] A data processing system capable of storing and / or executing program code may include at least one processor connected to storage elements directly or indirectly via a system bus. The storage elements may include local memory used during actual execution of the program code, mass storage, and cache memory in which at least a portion of the program code is temporarily stored to reduce the number of times the code is retrieved from mass storage during execution. Input / output or I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.) may be connected to the system either directly or through intermediate I / O controllers.

[0056] Network adapters can also be connected to the system to connect the computing system to other computing systems or remote printers or storage devices over private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0057] As used herein, the term "hardware processor subsystem" or "hardware processor" may refer to a processor, memory, software, or combinations thereof that work together to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or compute-element-based controller (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memory, etc.).In some embodiments, the hardware processor subsystem may include one or more memories that may be on-board or off-board, or that may be dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, BIOS (Basic Input / Output System), etc.).

[0058] In some embodiments, the hardware processor subsystem may include and execute one or more software elements. The one or more software elements may include an operating system and / or one or more applications and / or specific code to achieve a particular result.

[0059] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specific result. Such circuitry may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0060] These and other variations of a hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0061] Reference in the specification to "one embodiment" or "an embodiment" of the present invention, as well as other variations thereof, means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the present invention. Therefore, the phrases "in an embodiment" or "in an embodiment," as well as other variations appearing at various points in the specification, do not necessarily all refer to the same embodiment. However, it is to be understood that features of one or more embodiments may be combined in light of the teachings of the present invention contained herein.

[0062] It should be noted that the use of " / ", "and / or", and "at least one of", e.g., in the cases "A / B", "A and / or B", and "at least one of A and B", is intended to cover only the selection of the first listed option (A), or only the selection of the second listed option (B), or both options (A and B). Another example: In the cases "A, B, and / or C" and "at least one of A, B, and C", this wording is intended to cover only the selection of the first listed option (A), or only the selection of the second listed option (B), or only the selection of the third listed option (C), or only the selection of the first and second listed options (A and B), or only the selection of the first and third listed options (A and C), or only the selection of the second and third listed options (B and C), or all three options (A and B and C). This can be extended to any number of the listed options.

[0063] The foregoing is to be considered in all respects illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein should be determined not from the detailed description, but rather from the claims, which are to be construed to the full breadth allowed by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the present invention and that various modifications may be made by those skilled in the art without departing from the scope and spirit of the invention. Those skilled in the art may implement various other combinations of features without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the particulars and particulars required by the patent laws, what is claimed and intended to be protected by the specification is set forth in the appended claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 424,517

[0001] US 63 / 464,247

[0001] US 18 / 505,732

[0001]

Claims

[1] A method for tracking movement, comprising: performing (204) person detection in individual images from a plurality of video streams to identify detection images; Combining (206) visual and location-related information from the detection images to generate scores for pairs of detection images across the plurality of video streams and across individual images of the respective video streams; Creating (210) a pairwise recognition graph using the recognition images as nodes and the evaluation values ​​as weighted edges; tracking (212) the movement of a person based on a constrained answer set programming problem, wherein the constraints are determined based on matching scores and logical assumptions; and Performing (606) an action in response to the tracked movement. [2] The method of claim 1, further comprising synchronizing the plurality of video streams to identify temporal correspondences between frames of the plurality of video streams. [3] The method of claim 1, further comprising extracting the visual information based on a visual similarity between detection images. [4] The method of claim 1, further comprising extracting the location information based on a projection of two-dimensional coordinates into a three-dimensional environment for the capture images and determining a distance between the projected coordinates. [5] The method of claim 1, wherein generating the pairwise detection graph comprises determining edges between detection images from different frames of the same video stream and determining edges between detection images from different video streams at corresponding times. [6] The method of claim 1, wherein the action comprises generating a report for a healthcare professional to use in making decisions regarding treatment of a patient based on the tracked movement of the patient. [7] The method of claim 1, wherein tracking the movement of the person relates to movement within a healthcare facility, and wherein the plurality of video streams are generated by video cameras within the healthcare facility. [8] The method of claim 1, wherein combining the visual and location information comprises adding an output from a visual branch to an output of a location branch. [9] The method of claim 8, wherein the visual branch comprises processing the recognition images with a re-identification model. [10] A system for tracking movements, comprising: a hardware processor (710); and a memory (740) storing a computer program that, when executed by the hardware processor, causes the hardware processor to: performing (204) person detection in individual images from a plurality of video streams to identify detection images; Combining (206) visual and location-related information from the detection images to generate scores for pairs of detection images across the plurality of video streams and across individual images of the respective video streams; Creating (210) a pairwise recognition graph using the recognition images as nodes and the evaluation values ​​as weighted edges; tracking (212) the movement of a person based on a constrained answer set programming problem, wherein the constraints are determined based on matching scores and logical assumptions; and Performing (606) an action in response to the tracked movement. [11] The system of claim 10, wherein the computer program further causes the hardware processor to synchronize the plurality of video streams to identify temporal correspondences between frames of the plurality of video streams. [12] The system of claim 10, wherein the computer program further causes the hardware processor to extract the visual information based on a visual similarity between capture images. [13] The system of claim 10, wherein the computer program further causes the hardware processor to extract the location information based on a projection of two-dimensional coordinates into a three-dimensional environment for the capture images and to determine a distance between the projected coordinates. [14] The system of claim 10, wherein the computer program further causes the hardware processor to determine edges between detection images from different frames of the same video stream and to determine edges between detection images from different video streams at corresponding times. [15] The system of claim 10, wherein the action comprises generating a report for a healthcare professional to use in making decisions regarding treatment of a patient based on the tracked movement of the patient. [16] The system of claim 10, wherein the tracked movement of the person relates to movement within a healthcare facility, and wherein the plurality of video streams are generated by video cameras within the healthcare facility. [17] The system of claim 10, wherein the computer program further causes the hardware processor to connect an output from a visual branch to an output of a location branch to combine the visual and location information. [18] The system of claim 17, wherein the visual branch includes a re-identification model to process the recognition images. [19] A method for tracking movements in a healthcare facility, comprising: performing (204) person detection in individual images from multiple video streams in a healthcare facility to identify detection images; Combining (206) visual and location-related information from the detection images to generate scores for pairs of detection images across the plurality of video streams and across individual frames of the respective video streams; Creating (210) a pairwise recognition graph using the recognition images as nodes and the evaluation values ​​as weighted edges; tracking (212) the movement of a person based on a constrained answer set programming problem, wherein the constraints are determined based on matching scores and logical assumptions; and Generating (606) a report for a healthcare professional to make decisions regarding the treatment of a patient based on the tracked movement. [20] The method of claim 19, wherein generating the pairwise detection graph comprises determining edges between detection images from different frames of the same video stream and determining edges between detection images from different video streams at corresponding times.

Images