Distributed computing for real-time object detection and tracking
By dynamically allocating object detection and tracking tasks between resource-constrained devices, using the ad-hoc network to implement object detection and tracking in resource-constrained environments, the implementation of complex visual object detection and tracking on battery-powered devices is solved. The problem is to realize efficient power management and computing power allocation in resource-constrained environments.
Patent Information
- Application Number
- CN201880099954.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-12-03
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2038-12-03
AI Technical Summary
In resource-constrained environments, especially on battery-powered devices, there are difficulties in performing complex visual object detection and tracking tasks, especially those without a powerful GPU, CPU or TPU.
Using distributed computing methods, the object detection and tracking tasks are dynamically and self-organizedly allocated between multiple nodes, and the ad-hoc network is used to allocate power and computing power among resource-constrained devices to achieve object detection and Tracking function.
Without the need for centralized data center connections, object detection and tracking can be effectively performed in resource-constrained environments, reducing power consumption between devices, and suitable for scenarios where traditional networks cannot be connected.
Smart Images

Figure CN113168503B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to object detection and tracking in video data. Background Art
[0002] Object detection and tracking in video data is part of the technology baseline for many contemporary applications and services, including autonomous driving, remote robotic surgery, population counting, product recognition, anomaly detection, etc. In a typical setup, this technology can be deployed as an embedded solution, e.g., within a device such as a vehicle, or as a client-server application, e.g., with a client such as a mobile device sending raw data (e.g., images of video frames) to a cloud data center that then performs object detection. The best performing solutions for object detection are based on computationally intensive convolutional neural networks (CNNs).
[0003] The fastest CNN-based visual object detection platforms (e.g., Darknet, TensorFlow) can run reasonably sized general-purpose CNNs (e.g., Yolo, Faster R-CNN) in real time on some hardware systems (e.g., desktop computers with powerful processors such as NVIDIA Titan X graphics processing units (GPUs)). However, despite the continuous and rapid improvement in the capabilities of mobile devices, performing object detection in smartphones, tablets, or other small computers embedded in devices at the edge of the network (e.g., in robots, drones, sensors) remains a very difficult task.
[0004] In fact, running algorithms with the sophistication of modern visual object detectors on battery-powered devices will likely remain a challenge for the foreseeable future, especially for devices that do not contain powerful GPUs, central processing units (CPUs), or tensor processing units (TPUs) but are instead based on relatively cheap, standard, energy-conscious CPUs or off-the-shelf microcontrollers. Summary of the invention
[0005] Several of the techniques, devices, and systems described herein address this problem by utilizing distributed computing for performing object detection and / or object tracking, where detection and / or tracking is performed by several nodes (e.g., several nodes in an ad hoc network). This distribution of computing is random, self-organizing (ad-hoc), and can take into account the computing power of a particular device.
[0006] These techniques may be particularly interesting in resource-constrained environments where devices form dynamic ad-hoc networks and have resource constraints in terms of battery power and / or computing power. In such cases, the disclosed techniques allow the power consumed by object detection and tracking functions to be distributed among multiple devices.
[0007] The specific techniques for distributed object detection and tracking are described in detail below. According to several of these techniques, a first current frame from a series of frames is sent to a first node to detect a first object. After receiving object detection information for classifying and / or locating the first object in the first current frame in response, a second node is selected and a second current frame is sent to the second node for updated object detection. In addition, while waiting for the result of the updated object detection, two or more frames after the second current frame are sent to each corresponding tracking node. Object modeling information indicating the position and / or category of one or more objects derived from the previously received object detection information is also sent to each of the corresponding tracking nodes. Tracking information for the first object is received from each of the corresponding tracking nodes.
[0008] The technology described in detail below includes an example method for tracking the position of an object in a series of frames of a video stream, wherein the method includes the steps of: selecting a first node from a set of two or more nodes that can be used for object detection; and sending a first current frame from the series of frames to the first node for detecting the first object in the frame. The method proceeds by the following steps: receiving object detection information for the first object from the first node. After receiving the object detection information for the first object from the first node, the method proceeds by several additional steps. First, a second node is selected from a set of two or more nodes that can be used for object detection, and a second current frame from the series of frames is sent to the second node for updated detection of the first object. In addition, each of the two or more frames after the second current frame is sent to each corresponding tracking node, wherein sending each frame to the corresponding tracking node includes: selecting the corresponding tracking node from the set of two or more nodes that can be used for tracking. In various embodiments, object modeling information derived from the object detection information is also sent to each of the corresponding tracking nodes - the object modeling information can be sent with the frame sent to the tracking node, or can be provided separately to all available tracking nodes. Finally, tracking information of the first object for the frame transmitted to each of the tracking nodes is received from the respective tracking nodes.
[0009] The above method may be repeated one or more times. Therefore, in some embodiments, the above example method may further include the following steps: receiving updated object detection information for the first object from the second node; selecting a third node from a set of two or more nodes that can be used for object detection; and sending a third current frame from a series of frames to the third node for further updated detection of the first object. The method may also include the following steps: sending each of the two or more frames after the second current frame to each corresponding tracking node, wherein sending each frame to the corresponding tracking node includes: selecting the corresponding tracking node from a set of two or more nodes that can be used for tracking; sending updated object modeling information derived from the updated object detection information to each of the corresponding tracking nodes; and receiving tracking information for the first object of the frame sent to the corresponding tracking node from each of the corresponding tracking nodes. It will be understood that the technique can be repeated until, for example, it is no longer necessary or possible to track the object.
[0010] In the following discussion, devices and systems corresponding to the above-described methods and their variations will also be described in detail.
[0011] At least in some embodiments, the disclosed techniques, devices, and systems can provide several advantages. For example, in general, these techniques do not require a connection to a centralized data center, which means that detection can be performed between nearby devices even when traditional network connections (e.g., IEEE 802.11x or cellular networks) are not available. Low-power, short-range network protocols such as Bluetooth or IEEE 802.15.4-based network stacks (XBee, 6LowPAN, etc.) can be used instead. Another advantage is that these techniques can work in resource-constrained environments where the computing performance of the device is low and / or the power consumption of the device is limited. Some embodiments of these techniques can be deployed using existing technologies and standards.
[0012] The technology, devices and systems disclosed herein are not limited to those summarized above. Variations of these technologies and additional advantages and uses of these technologies will be described below. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The embodiments and further objects and advantages thereof may best be understood by referring to the following description taken in conjunction with the accompanying drawings, in which:
[0014] Figure 1 Example configurations of nodes and roles are shown in accordance with some disclosed embodiments.
[0015] Figure 2 An example of interaction between a client node and a compute node is shown.
[0016] Figure 3 is a signaling flow diagram according to some embodiments of the presently disclosed technology.
[0017] Figure 4 An example of identifying candidate nodes according to some embodiments is shown.
[0018] Figure 5 Example techniques for distributed object detection and tracking are shown.
[0019] Figure 6 An example of segmenting an image for distributed object detection is shown in accordance with some embodiments.
[0020] Figure 7 is a process flow diagram illustrating an example method according to some embodiments.
[0021] Figure 8 is a block diagram of an example device according to some presently disclosed embodiments.
[0022] Fig. 9 A computer program based implementation according to some embodiments is shown. DETAILED DESCRIPTION
[0023] In the coming years, it can be expected that mobile devices will be able to run increasingly accurate object detection algorithms. However, as mentioned above, running algorithms with the sophistication of modern visual object detectors on battery-powered devices will remain a challenge for the foreseeable future, especially for devices that do not contain powerful GPUs, CPUs, or TPUs.
[0024] Meanwhile, in contrast to object detection, object tracking may be feasible on various devices because the complexity of object tracking is less than that of object detection.
[0025] A possible alternative to the client-server model (where the server performs both object detection and tracking tasks) is a hybrid solution that uses a combination of server-based object detection and client-based object tracking to provide an object detection solution that is more accurate than simple object tracking but does not require running a computationally expensive object detector on the client. However, this approach still requires access to the server.
[0026] Several of the techniques, devices, and systems described herein address this problem by utilizing distributed computing for performing object detection and / or object tracking, where detection and / or tracking is performed by several nodes (e.g., several nodes in an ad hoc network). This distribution of computing is random, self-organizing (ad-hoc), and can take into account the computing power of a particular device.
[0027] These techniques may be particularly interesting in resource-constrained environments where devices form dynamic ad-hoc networks and have resource constraints in terms of battery power and / or computing power. In such cases, the disclosed techniques allow the power consumed by object detection and tracking functions to be distributed among multiple devices.
[0028] Use cases that may be particularly well suited for these techniques may include, for example:
[0029] - Multiple drones flying over the same industrial object (or performing rescue missions), where their mission requires object detection.
[0030] - A team of field engineers in a factory (or in a pipeline, or working on a power grid, etc.) who need visual object detection on their mobile phones for augmented reality support applications.
[0031] - Multiple devices operating in remote environments (e.g., mines, war zones, etc.) where connectivity to a server is unavailable or very expensive.
[0032] The techniques described herein are suitable for implementation in a network consisting of a plurality of nodes that are interconnected and reachable in the network, which may be an infrastructure network, a dynamically created mesh network, an ad-hoc network, or some combination of the two.
[0033] Figure 1 Two example configurations of object detection and / or tracking systems that can utilize the currently disclosed technology are shown. The nodes shown, including client ("cl") nodes 110, computing ("cmp") nodes 130, master ("mst") nodes 120, and general ("nd") nodes 140, should be understood as logical nodes, or "roles", because a given device can act according to different roles in these roles at different times, and in some cases, as further described below, can act according to two roles at a time. That is, it should be understood that each of the computing nodes 130 and general nodes 140 shown will generally correspond to a unique device connected to other devices via an ad hoc or conventional network.
[0034] During an object detection and / or tracking session, Figure 1 The nodes shown perform the roles described below.
[0035] Client ("cl") role: This role is played by the node that requests to perform object detection and / or object tracking tasks. Figure 1 The cl node in 130 is responsible for providing the raw input material to the compute ("cmp") node 130, either directly (e.g. Figure 1 configuration (a)) or via the master ("mst") node 120 (as shown in FIG. Figure 1 The original input material includes image data or video data encoded in a certain format (for example, JPEG, PNG, BMP, TIFF, GIF in the case of images, and MPEG-4, H.264, H.265 in the case of videos).
[0036] Compute ("cmp") role: A node with this role performs object detection and / or object tracking tasks according to assignments received from the master ("mst") node 120. Figure 1 The object detection service implemented by the cmp node 130 is considered to be stateless because it does not require storage - along with the assignment of tasks, the necessary raw input material for providing the task is provided. A specific example of an object detection task is provided below.
[0037] Master ("mst") role: The node with the master role is responsible for assigning compute roles to compute ("cmp") nodes. Figure 1 These assignments are dynamic, i.e., the mst node 120 can reallocate computing roles, for example, based on current cmp node performance or if a cmp node 130 leaves the network. Figure 1 As shown in configuration (b), the MST node 120 can be used as an endpoint for raw data traffic from the client ("CL") node 110. In this case, the MST node 120 accepts all images or video frames from the C1 node 110 and selects a CMP node to which all images or video frames are forwarded. In this case, the MST node can also be viewed as performing load balancing tasks while also striving to provide high availability of appropriate computing power via the CMP node 130.
[0038] "nd" role: For completeness, this node is Figure 1 ND node 140 is shown in the figure and plays a general role corresponding to a node that does not actively perform object detection and / or object tracking services but is adjacent to and visible to other nodes. These nodes are not eligible to be MST and / or are not selected by MST to be used as CMP nodes.
[0039] As described above, the roles are logical. A physical device can transition from one of the roles shown to another, for example, from "cmp" to "nd" and vice versa. For example, a physical node can also have both an mst role and a cmp role. Further, a physical node can support multiple object detection sessions, and in some embodiments, the physical node is, for example, a client node (cl) for one session and a master node (mst) and / or a compute node (cmp) for another session.
[0040] exist Figure 1 , configuration (a) can be understood as an embodiment in which the MST node 120 acts as a computing coordinator, while configuration (b) shows an embodiment in which the MST node 120 acts as a computing endpoint. As shown in the figure, this means that the raw data and object detection / tracking results are passed directly between the C1 node 110 and the CMP node 130 in the former configuration, while in the latter both pass through the MST node 120. As can be seen from the figure, in configuration (a), the MST node 120 selects the CMP node 130 ("node selection" or "node selection") and manages the CMP node 130, but does not process the raw data and object detection / tracking results. On the other hand, in configuration (b), everything passes through the MST node 120, so that the C1 node 110 does not need to communicate directly with the CMP node 130 at all.
[0041] Figure 2 An example of interaction between the c1 node 110 and the cmp node 130 is shown. As shown in the figure, the c1 node 110 Figure 1 (a) sends the original input material directly to the cmp node 130 according to the instructions from the mst node 120 acting as the coordinator, or Figure 1 In configuration (b), the original input material is sent to the cmp node 130 via the mst node 120 acting as a computing endpoint. For the object detection task, in the case of video data, the cmp node 130 extracts frames from the raw material and detects one or more objects in the frames, and also returns object detection information to the cl node directly or via the mst node 120. The object detection information includes, for example, identification of one or more objects based on the type of the object, and information indicating one or more positions of the object in each frame in the frame. In the example shown, the object detection information returned to the c1 node 110 is serialized in JSON representation. However, other formats or representations of object detection information are possible. For example, the object detection information may include an image or video with object recognition information superimposed on the image or video, such as Figure 2 In the case where multiple frames are assigned to the cmp node 130 for a given object detection task, the detected objects are sent back to the cl node 110 frame by frame. Example code for the detected objects may include:
[0042]
[0043]
[0044] In addition to roles, it can also be assumed that each physical node in the network has some basic properties. First, each physical node has an identifier, such as a media access control (MAC) address corresponding to the wireless network interface of the physical node. This identifier can be transmitted to other nodes upon request and used when selecting an MST or CMP node. Secondly, each physical node has a storage device (storage) in which information corresponding to other physical nodes in the network is stored. For example, one or more reputation scores can be stored for other nodes for selecting the best available CMP node or MST node, where these reputation scores reflect the performance of the node as an MST node or CMP node in previous object detection sessions.
[0045] In some embodiments, the MST roles may be assigned to nodes in the network, for example, by voting based on the reputation of the nodes in the network or randomly, or may be pre-selected / hard-coded by the implementer of the distributed object detection / tracking system. In some embodiments, the MST node 120 is also capable of delegating responsibilities in a manner that effectively switches the processing model from a processing model for distributed computing for both object detection and object tracking as described herein to a conventional client-server mode, or to a client-server mode in which object detection is performed by one or more servers and object tracking is performed in a distributed manner by the CL node 110 and / or the CMP node 130. For example, through this approach, if the above-mentioned collection of drones suddenly gains access to such computing resources, each drone can update the "resource table" maintained by the drone with an entry corresponding to the one or more servers currently available, in addition to other entries corresponding to the CMP nodes 130 accessible to the drone, wherein the new entry reflects the higher computing resources and power resources of the one or more servers relative to other CMP nodes 130 in the ad-hoc network of drones. Then, if connectivity to that server is subsequently lost, all drones can update their "resource tables" by removing that entry, retaining only entries for other available cmp nodes 130 in the network.
[0046] In some embodiments, nodes in the network store two or more types of reputations as value pairs, as described in further detail below, where these value pairs are used to select a master node for a given session and to select computing nodes for performing object detection and / or object tracking tasks during the session.
[0047] First, each of the physical nodes in the<identifier,mst_reputation_index> The value pair holds the reputation of other physical nodes in the network relative to the role of the mst (master node). The identifier value represents a unique identifier for a given physical node, such as a MAC address. The mst_reputation_index value reflects the performance of the node as an mst node 120 in previous object detection sessions.
[0048] Calculate the mst_reputation_index (or mst_rep in mathematical terms) of the identifier identifer ) can be as follows:
[0049] -Let mst_dsr identifer is the ratio of completed object detection and / or tracking sessions to the total object detection and / or tracking sessions that the node participated in as an MST node 120. (For example, if a node acting as an MST node 120 leaves the network before a C1 node 110 closes / completes an object detection and / or tracking session, this value will be less than 1).
[0050] -Let mst_cmpreselect identifer is the ratio of initial CMP appointments made by the node acting as MST to the CMP re-elections that had to be made (this value will be less than 1 if the CMP node selected by the node acting as MST leaves the network before the C1 node 110 closes / completes the object detection session).
[0051] - Let mst_battery_remaining be a normalized 0 to 1 value of the remaining battery on the node (eg, where a value of "1.00" indicates a full battery, while values close to 0 indicate an impending shutdown).
[0052] -Then, with respect to its performance as an MST node 120, the reputation index of the node may be a weighted average as shown below, where higher values are better:
[0053]
[0054] The reputation index of the mst node 120 may be calculated and updated on the c1 node 110, for example, where each c1 node 110 sends a keep-alive message to the mst node 120 to find out whether the node is still active. In addition, when the mst node 120 performs a reselection of the cmp node 130, it may notify the c1 node 110 of it so that the index may be updated.
[0055] akin<identifier,cmp_reputation_index> The value pair can be used to store the reputation of each of the other physical nodes in the network regarding their performance in the role of a cmp (compute) node. Similarly, the identifier value represents a unique identifier for a given physical node, such as a MAC address. The cmp_reputation_index value reflects the performance of the node as a cmp node 130 in previous object detection sessions.
[0056] As with the mst_reputation_index value discussed above, a cmp_reputation_index value may be calculated to give a value for the reputation of a physical node with a given identifier in the role of cmp node 130 (cmp_reputation_index). identifier ). This helps calculate cmp_rep identifier An example of a variable may be the ratio of completed object detection sessions to total object detection sessions in which the node participated as a cmp node 130. identifier It can be an indication of the performance of the physical node as a cmp node 130 in terms of the total number of frames processed and sent to the cmp node 130. Similarly, the same weighted average method can be used to calculate the reputation index. The same battery remaining parameter can also be used here. In some embodiments, a single cmp_reputation_index can be calculated and stored for each of several physical nodes in the network, corresponding to all computing tasks performed by the physical node. In other cases, separate scores can be calculated for different types of computing tasks, for example, separate scores for object detection and object tracking tasks.
[0057] The performance of cmp node 130 may in some embodiments be performed by a physical node having an mst role (e.g., in the case where mst node 120 is used as an endpoint for computation, such as Figure 1 ) or by a physical node with a cl role (e.g., in the case where the mst node 120 is used as a computing coordinator, as shown in (b) in FIG. Figure 1 , as shown in (a) above.
[0058] Figure 3Two versions of an example object detection process according to some techniques disclosed herein are shown, wherein the first version corresponds to a configuration in which the MST node 120 operates as an endpoint, and the second version corresponds to a configuration in which the MST node 120 acts as a computation coordinator. Signaling designated as 310 may be present in either version, while signaling 330 and 350 correspond to the cases of "MST as an endpoint" and "MST as a computation coordinator", respectively.
[0059] The object detection process is triggered by a request from a client (c1), which may start the process by selecting an mst node 120 from available nodes or randomly selecting a node. Alternatively, in some embodiments, the mst node 120 may be known a priori rather than dynamically selected.
[0060] If the mst node 120 is selected dynamically, then Figure 3 As shown, the process can begin by broadcasting a request for object detection, as shown at 312. For example, the broadcast can use any of a variety of broadcast or flooding techniques for distributing the broadcasted request among nodes of a mesh network or ad hoc network. The broadcasted request can include specific requirements for the service, such as what format type to use for the original material. It can be encapsulated, for example, by using a MIME type (or media type as it is also known). Here is an example of how an MPEG-4 video stream encapsulated in an exemplary JSON object might look:
[0061] {
[0062] "mtype":"video / mp4"
[0063] }
[0064] If it was an image instead of a video, you could use something like this:
[0065] {
[0066] "mtype":"image / jpeg"
[0067] }
[0068] As in Figure 3As shown at 314 in FIG. 1 , a selection process (commonly referred to as a “selection” process) may be implemented among the nodes that received the broadcasted request. This process may be accomplished, for example, using any of the well-known leader election algorithms used in distributed systems in general and wireless networks in particular. For example, some of these algorithms are described in S. Vasudevan, B. DeCleene, N. Immerman, J. Kurose, and D. Towsley, “Leader election algorithms for wireless ad hoc networks,” Proceedings DARPA Information Survivability Conference and Exposition, 2003, Vol. 1, pp. 261-272.
[0069] A common aspect in leader selection algorithms is weights, internal variables that contain the current value of the criteria used to select a leader between nodes. In some embodiments of the currently disclosed system, the "weights" can be based on the "reputation index" discussed above. Another common aspect of leader selection is the identification of the node, because each node in the network has a unique identification that it can provide upon request. In embodiments of the currently disclosed system, the identification can be, for example, the MAC address of the wireless network interface of the device.
[0070] An alternative to using a leader election algorithm is a simple random selection by the cl node 110. Another alternative is to have the mst role assigned a priori, where the c1 node 110 has been previously informed of the assignment. Figure 3 As shown at 316 in , in this case, the c1 node 110 simply contacts the mst node 120 directly to initialize the object detection process.
[0071] Given the selection of the mst node, the cmp node 130 is assigned to perform the task of object detection. Figure 3 In some embodiments, the algorithm for selecting the cmp node 130 has the following steps:
[0072] Step 1: First, identify candidate nodes in the network. Figure 4An example is shown in which an example network is shown in the form of a graph, where vertices indicate candidate nodes and edges represent connections between these nodes. In this example network, the MST node 120 is directly connected to nodes A and B, and indirectly connected to other nodes. This may mean, for example, that A and B are within the range of the transceiver in MST mode, while the other nodes are not. In this example, in order for the MST node 120 to reach node C, node B or nodes A and D need to relay their messages.
[0073] To identify eligible nodes, one approach is to perform a simple graph traversal process from the mst node 120 to all other available nodes in the network. Since the network may be large, in some embodiments, this process may be limited by two factors:
[0074] When encountering the cl node 110, terminating the traversal of the current path in the graph; and
[0075] When the number of hops reaches a threshold (e.g. 5), stop traversing the graph.
[0076] The graph traversal process can be performed using a simple identification request, which can be sent from one node to another by broadcasting to all nodes in the coverage area. Each node responds to the request using at least its identifier, where the response is relayed back to the MST node. The request is propagated by the receiving node further down the network. The resulting "walk" of the network is used to identify nodes that can be reached from the MST node 120 (in the case of the "MST as a computing endpoint" configuration) or from the CL node 110 (in the case of the "MST as a coordinator" configuration) or from both.
[0077] Step 2: In some embodiments, a first selection process is implemented to select a cmp node 130 for performing the object detection task from the identified candidates. As described above, the selection can be made based on the reputation index of each of the nodes. In this case, the mst node 120 collects the reputation of all cmp-eligible nodes and selects the most reputable node. There may be some physical nodes that have not yet assumed the cmp role, so there is no reputation record. In this case, the selection of eligible cmp nodes 130 is completed among those physical nodes for which the cmp reputation index exists and is above an acceptable threshold. If there are not enough physical nodes with a cmp reputation index or the reputation index is below a certain threshold, the cmp node 130 can be selected based on the distance between the cmp node 130 and the cl node 110 (for example, see Figure 4, nodes C, then B and D may be selected first because they are closer to the cl node 110). Finally, if all nodes are at an equal distance, the cmp node 130 may be randomly selected from the list of available nodes.
[0078] return Figure 3 Once the cmp node 130 and the mst node 120 have been selected, the object detection process can begin. Object detection can be performed on a per-frame basis or on a per-group-of-frame basis, where different frames or groups of frames are sent to different cmp nodes 130. Figure 1 (a)) or as a computation endpoint ( Figure 1 (b)) in the figure comes from the c1 node 110 or the mst node 120 to make the decision to split the frame between nodes.
[0079] exist Figure 3 The basic building blocks of the algorithm are shown in , where signaling 330 corresponds to the case of "CMP as endpoint" and signaling 350 corresponds to the case of "CMP as coordinator". The cl node 110 sends raw material including one or more frames or videos to the selected cmp node 130 via the mst node 120 or directly, as shown at 336 and 356. The selected cmp node 130 performs object detection and returns the detected object categories, their positions on the frame, and frame identifiers, as shown at 338 and 358.
[0080] The process shown may utilize a distributed object detector-tracker algorithm based on the following two assumptions:
[0081] - First, it can be assumed that updates from object detectors are more reliable than updates from object trackers, since the latter use simpler algorithms. However, object trackers are typically much faster than object detectors. Therefore, as object detector frames become available, they are used to correct the model and position of the object(s) of interest.
[0082] - Second, the tracker is faster than real time, for example, capable of processing more than 60 frames per second. This is in line with current state-of-the-art trackers, many of which process more than 200 frames per second.
[0083] A version of the algorithm for one class of objects can be described as follows. The parameter θ (theta) represents the number of frames that the tracker can process between any two consecutive frames of the video sequence (i.e., before it needs to send the next frame for visualization). In the subsequent discussion, frame "k" in the frame sequence is denoted as f K , and b Krefers to the bounding box of a given object class for frame “k”, where b K = = [x k ,y k ,w k ,h k ]. A bounding box defines a window for the detected location of objects of a given object class. In this example, (x, y) is the center of the bounding box, and (w, h) are its width and height. Other representations (e.g., polygons) are also possible.
[0084] For each frame f K , used to track the object M k The model of is composed of feature vectors. These vectors are extracted from the previous frame region representing the object of interest (i.e., the bounding box returned from the previous frame).
[0085] The feature vector has the following format: and contains multiple elements about the object, which in our example is a region of interest in the image that describes the class of object (e.g., "car"). What the elements represent depends on the implementation. For example, in one implementation, they may represent raw pixel intensities (typically red-green-black, or RGB intensities), in another implementation the average of each of the color channels, in another implementation the color mean and standard deviation, and in another implementation a color histogram indicating the distribution of colors in the image. There may be many of these vectors that make up the model, e.g., a representation of all previous encounters of an object in previous frames.
[0086] During the tracing process, function T converts object B k-1 The previous bounding box is mapped to the current bounding box B k :
[0087] B n =T(f n ,M n ,B n-1 )
[0088] The implementation of T uses a vector similarity measure, for example, cosine similarity or cross-correlation.
[0089] Some embodiments of the presently disclosed technology may utilize asynchronous communication between the cl node 110 and the cmp node 130 providing object detection services. According to this approach, multiple frames may be tracked while object detection is being performed. This results in the following processing of the frames:
[0090] - If only tracking information is available for the most recent frame (detection is not available), simply use the feature vector from the last available frame to update the model
[0091] - If object detection from the previous frame (nk) is available, where n is the current frame and k is the number of tracked frames between the detected frame and the current frame:
[0092] o If k is less than or equal to θ (meaning the tracker can process all k frames before the next frame of the video sequence or image frame is passed), then n Remove the eigenvector X n-1 , X n-k+1 , the detected object position from frame f n-k Propagate to f n , and recursively update the model o. If k is greater than θ (meaning that the tracker cannot process all k frames before the next frame of the video sequence or image frame is incoming), these frames are subsampled, for example by iterating the same process as above on every other frame, thereby reducing the complexity of the tracker.
[0093] An example of a distributed object detection and tracking algorithm that can be used with the above-described network configuration and process is described below. The algorithm works by processing each frame independently. A "frame" here refers to an image, which can be part of an image stream, or extracted from a video stream. Depending on whether the MST node 120 is acting as an endpoint or simply a computation coordinator, the frame is extracted from the video stream at the MST node 120 or at the C1 node 110, respectively. The same is true regarding which node controls the overall progress of the algorithm. For the purposes of discussion, the entity running the distributed version of the detection and tracking algorithm is denoted as the controlling entity or "CE". This can be a physical node with an MST role (e.g., where the MST node 120 is in the Figure 1 configuration (b) is an endpoint), or a physical node with a cl role (for example, where the mst node 120 is in Figure 1 Configuration (a) is the computation coordinator).
[0094] For the purpose of this discussion, it is assumed that CE already knows the number of CMP nodes 130 allocated by MST node 120 for the computation before the object detection and tracking process begins. In some embodiments, each CMP node 130 can be parameterized by at least two parameters:
[0095] - Its computational power, which represents how quickly it can process incoming frames for object detection or object tracking.
[0096] - Its remaining battery life, if the node uses renewable power.
[0097] The process Figure 5. The process begins with CE selecting one of the eligible CMP nodes 130 (e.g., the CMP node 130 selected by MST node 120 as described above) for object detection of the first incoming frame. The selection is constrained by the parameters described above, such as so that the node with the best computing power and / or battery life is selected. There can be a bias in the selection criteria (e.g., 70% computing power and 30% battery life) to reduce the chance of multiple candidates having the same score.
[0098] Before CE receives the result returned by each first object detection, it cannot select other cmp nodes 130 for tracking. Figure 5 The next frame f is sent to the cmp node 130 selected for object detection (which may be the same or different from the node that performed the first object detection). k+1 ,like Figure 5 As shown. Between this time and the receipt of the next result of object detection (at "df2"), the ce node receives multiple frames. These frames are submitted to the remaining cmp nodes 130 in the eligible nodes for tracking. The submission of these frames to the cmp nodes 130 selected for these tracking tasks includes object detection information from the most recently detected frames (e.g., for those that are f1) to assist in tracking. When the tracked frames are completed and received, the updated position of the object can be submitted for use in subsequent tracked frames.
[0099] From the perspective of the ce node, the process is described in more detail as follows. Let f n is the current frame in the queue. Assume that there are at least two qualified cmp nodes 130; in general, there may be more. Let C = (<cmp1,rep_cmp1> ,<cmp2,rep_cmp2> ,…, <cmp N ,rep_cmp N >) is a set of cmp nodes 130 that meet the conditions. The second set C tracking Store the nodes that are currently tracking frames. For simplicity, it can be assumed that all eligible cmp nodes 130 (even resource-constrained cmp nodes 130) have the ability to track at least one frame before the next frame becomes available.
[0100] For each incoming frame f n
[0101] - If an object detection frame previously issued for object detection is still being processed (i.e., if ce has not yet received object detection information for a previously submitted frame):
[0102] ο Find out how many cmp nodes 130 have completed the trace and remove them from C tracking Removed;
[0103] o If tracking results are received out of order, reorder the tracked frames as needed;
[0104] οFrom C\{cmp X ,C tracking}, the cmp node 130 with the highest calculated reputation index is selected, provided that its battery can track more frames. If not, the node is completely removed from C, and the next node is selected for tracking, again, provided that its battery can track more frames.
[0105] οSend frame f to the selected node n For tracking, the selected nodes are added to C tracking .
[0106] - If frame f k The object detection frame just becomes available to the ce node:
[0107] o Update the model as described above and send the updated model to the cmp node 130 for use in subsequent tracking tasks. This means that the model M is brought up to date with the detected frame fk so that the latest detection of the object position can be used in subsequent tracking tasks. The updated model can be broadcast to all cmp nodes 130 in C so that each eligible node has the updated model if necessary, or the updated model can be submitted to each cmp node 130 together with the raw data to be processed by the cmp node 130 when each cmp node 130 is selected for a tracking task.
[0108] o Select the cmp node 130 with the highest calculated reputation index from C (e.g., cmpX∈C), provided that its battery can detect more frames. If not, the next node can be selected, provided that its battery can detect more frames. Note that if a cmp node 130 is discarded due to insufficient battery to support the detection task but it still has sufficient battery for tracking, it can be left in C; otherwise, it can be removed from C entirely.
[0109] o Send frame fn to cmpX for detection. Frame fn may also be sent to another node for tracking.
[0110] In the above discussion, it is assumed that the selection of the MST node 120 for controlling the object detection and / or tracking task is used. The use of the MST node 120 is suitable for the case where the device has computing resources and enough power to assume the role of the coordinator. The device can be, for example, an Internet of Things (IoT) gateway. There can also be a situation where all nodes are the same (e.g., a sensor ad-hoc network / mesh network). In this case, as described above, the MST node 120 can be used, but the distributed object detection and tracking task can also be implemented without a master node. With this method, all nodes (from the above use case) can broadcast their battery status and the resources available for detection to all other nodes at specific time intervals (e.g., every minute and also in response to state changes). In this way, all nodes can maintain a similar "resource table", and if one of these nodes becomes a client (needs to detect a visual object), it uses the table to send a request to the most suitable node (e.g., a node with the most battery or computational complexity resources). In order to obtain optimal performance, the "resource table" should be updated in all other nodes at the same time point to reflect the selection of nodes for detection or tracking tasks.
[0111] Other variations of the above techniques are possible. Above, an algorithm for distributing calculations for object detection and tracking methods among multiple cmp nodes 130 is described in detail. As described above, it is assumed that the object detection task for a given frame is assigned to a single cmp node 130. This may have a negative effect, namely, in some cases, the accuracy of the detection may decrease due to the time between two subsequent object detections, such as when there are several objects to be detected in one frame. This decrease may be particularly noticeable if one or more objects being tracked are temporarily outside the field of view of the camera. In this case, the tracker will lose the object and may not be able to find it again until the next object detection frame happens to include the object.
[0112] In order to reduce the time spent processing a frame for object detection, the frame may be segmented into multiple parts and the parts may be sent for detection to different cmp nodes 130. As described above, the selection of a cmp node 130 to send the parts may be accomplished using the same process as used to select a cmp node 130 or tracking node.
[0113] Figure 6 shows an example of how a frame can be segmented. The internal bounding box in each figure (from Figure 6The (a) unsegmented frame starts with) indicates the coordinates of the last tracked frame before receiving a new object detection frame, which should produce a more accurate estimate of where the object is. The outer bounding box is an estimated margin, which is an extrapolation of the inner bounding box. Based on the implementation, this margin can be larger or smaller than shown here.
[0114] Assume there are two parts, Figure 6 (b) and Figure 6 (c) in Figure 3 shows two possible ways to segment based on the horizontal and vertical axes of the yellow bounding box. Depending on the availability of the cmp node, the horizontal and vertical axes can also be combined to split the image into 3, 4, 5, 6 or more segments.
[0115] If this segmentation method is used, the above algorithm is changed to reflect that the object detection results of the frames segmented for object detection may arrive at different times. Therefore, actions that rely on object detection information of the received frame must be delayed until all parts are received. The object detection frame can then be reconstructed based on the object detection information received for the various parts. Likewise, the algorithm is modified to account for the fact that each frame to be sent out for object detection must be segmented and sent to multiple cmp nodes 130, which are selected according to a process similar to that described above.
[0116] In view of the detailed examples and alternatives discussed above, it will be appreciated that Figure 7 The process flow diagram of shows a generalized method for tracking the position of an object in a series of frames of a video stream according to some currently disclosed techniques. As described above, the method can be implemented by a device acting as an MST node 120 and / or a control entity, or can be implemented by a C1 node 110 acting alone, or by a combination of two or more devices working together. Please note that the use of the terms "first", "second", etc. in the following discussion is not intended to indicate a chronological or other order, but is instead used to distinguish one thing from another similar thing.
[0117] As shown at box 710, the method shown begins by selecting a first node from a set of two or more nodes that can be used for object detection. As shown at box 720, the method continues to send a first current frame from a series of frames to the first node to detect at least a first object in the frame. In some embodiments, the first node is selected according to the above-mentioned scoring process. The sending of the first current frame corresponds to the sending of the above-mentioned "raw material", wherein the raw material may include one or more digital images, a portion of a video, etc. It will be understood that, for example, if the image is segmented as described above, the term "frame" as used herein may refer to a portion of a digital image.
[0118] Object detection information for the first object is then received back from the first node, as shown in block 730. (It will be appreciated that several intervening frames may be processed during this period—for simplicity, their processing is omitted from this portion of the illustrated process.) The object detection information includes information that classifies and / or locates the first object in the frame previously sent to the first node for the object detection task. Of course, in some cases, object detection information for more than one object may be received.
[0119] Once the object detection information is received back from the first node, object tracking of the first object may be performed in one or more subsequent frames based on the object detection information. In addition, a new frame may be sent out for object detection. Therefore, as shown at block 740, the method continues to select a second node from a set of two or more nodes that can be used for object detection, and send a second current frame from a series of frames to the second node for updated detection of the first object, as shown at block 750. Note that the second node may be selected according to the selection process described above, and the second node may or may not be the same as the first node.
[0120] In addition, as shown at block 760, each of the two or more frames following the second current frame is sent to each corresponding tracking node. Here, sending each frame to the corresponding tracking node includes: selecting the corresponding tracking node from a set of two or more nodes available for tracking, as shown at block 762. In addition, as shown at block 770, object modeling information indicating the location and / or classification of one or more objects is sent to each of the corresponding tracking nodes together with or separately from the sending of the corresponding frame to each node (e.g., as a broadcast to all eligible nodes), the object modeling information being derived from the object detection information. Finally, as shown at block 780, tracking information for the first object of the frame sent to the node is received from each corresponding tracking node. The information can be reordered (if necessary) to resolve any information received out of order and processed for rendering / viewing.
[0121] The process shown may be repeated. Thus, for example, after sending two or more frames following the second current frame to each respective tracking node, the method may proceed to receiving updated object detection information for the first object from the second node (block 730); selecting a third node from the set of two or more nodes available for object detection (block 740); sending a third current frame from the series of frames to the third node for further updated detection of the first object (block 750), etc.
[0122] Thus, for example, after sending the third current frame to the third node, the method may continue to send each of the two or more frames following the second current frame to the respective tracking nodes, wherein sending each frame to the respective tracking nodes includes: selecting the respective tracking node from the set of two or more nodes available for tracking (blocks 760 and 762), in which case, sending updated object modeling information derived from the updated object detection information to each of the respective tracking nodes (block 770). Receive tracking information for the first object of the frame from each of these respective tracking nodes (block 780), and continue the process as needed.
[0123] In some embodiments, sending each of the two or more frames after the second current frame to each respective tracking node includes sending each frame between the second current frame and the third current frame to the tracking node. In other embodiments, such as when there are not enough nodes available for tracking or tracking resources are otherwise limited, fewer than all intervening frames (e.g., every other frame) may be sent for tracking.
[0124] As described above, in some embodiments, sending the object modeling information derived from the object detection information to each of the respective tracking nodes includes sending the object modeling information and the corresponding frame to each of the respective tracking nodes. In other embodiments, sending the object modeling information derived from the object detection information to each of the respective tracking nodes includes sending the object modeling information to all nodes in a set of two or more nodes available for tracking, such as via a broadcast or flooding process.
[0125] In some embodiments, Figure 7 The method shown may also include maintaining a single set of nodes that can be used for detection or tracking, wherein the maintenance includes removing the node from the set as each node selected for detection or tracking performs a corresponding detection or tracking, and returning the node to the set as each node selected for detection and tracking completes its corresponding detection or tracking. In other embodiments, different sets of nodes that can be used for detection and tracking may be maintained separately, wherein the maintenance includes removing the node from the corresponding set as each node selected for detection or tracking performs a corresponding detection or tracking, and returning the node to the corresponding set as each node selected for detection and tracking completes its corresponding detection or tracking.
[0126] In some embodiments, selection of a node for object detection is based on one or more metrics of each node in a set of two or more nodes available for detection, wherein the one or more metrics include or are based on any of: a battery status of the corresponding node; a measurement of processing resources available at the corresponding node; a count of previous object detection tasks completed by the corresponding node; and a ratio of object detection tasks completed by the corresponding node to the number of object detection tasks assigned to the corresponding node.
[0127] Similarly, in some embodiments, selection of a node for object tracking is based on one or more metrics of each node in a set of two or more nodes available for tracking, wherein the one or more metrics include or are based on any of: a battery status of the corresponding node; a measurement of processing resources available at the corresponding node; a count of previous object tracking tasks completed by the corresponding node; and a ratio of object tracking tasks completed by the corresponding node to the number of object tracking tasks assigned to the corresponding node.
[0128] Figure 8 1 is a schematic block diagram showing an example of a device 120 based on a combination of processors 122, 123 and hardware circuits 124, 125 and combined with a suitable memory unit 121. Therefore, the overall functionality is divided between the programming software for execution on one or more processors 122, 123 and one or more preconfigured or possibly reconfigurable hardware circuits 124, 125. The actual hardware-software partitioning can be determined by the system designer based on numerous factors, including processing speed, implementation cost and other requirements. The device 120 can be configured with appropriate program code in, for example, memory 121 and / or appropriate configuration of hardware circuits 124, 125 to perform any of the techniques described herein. Therefore, for example, the device 120 can be configured to act as a C1 node 110, a CMP node 130, an MST node 120 or a CE node, and in some embodiments can be configured to selectively play any of several of these roles.
[0129] In some embodiments, the device 120 is also configured with appropriate program code in the memory 121 and / or appropriate configuration of the hardware circuits 124, 125 to perform Figure 7 In some embodiments, two or more of the processing steps shown in FIG. Figure 8 The devices shown may be configured to cooperate with each other to perform Figure 7 The method shown or its variants.
[0130] Fig. 9Implementations based on computer programs according to some embodiments are shown. In this specific example, at least some of the steps, functions, processes, modules and / or boxes described herein are implemented with a computer program 240, which is loaded into a memory 220 for execution by a processing circuit including one or more processors 210. The processor 210 and the memory 220 are interconnected to each other to implement normal software execution. An optional I / O unit 230 can also be interconnected to the processor 210 and / or the memory 220 to implement input and / or output of relevant data (e.g., video frames and detection information).
[0131] The term "processor" should be interpreted in a general sense as any circuit, system or device capable of executing program code or computer program instructions to perform specific processing, determination or computing tasks.
[0132] Thus, the processing circuitry including the one or more processors 210 is configured to perform well-defined processing tasks such as described herein when executing the computer program 240 .
[0133] The processing circuitry is not necessarily dedicated to performing only the above-mentioned steps, functions, processes and / or blocks, but may also perform other tasks.
[0134] In one embodiment, the computer program 240 includes instructions that, when executed by the at least one processor 210, cause the at least one processor 210 to perform Figure 7 All or some of the steps shown in the processing flow.
[0135] The proposed technology also provides a carrier 250 comprising the computer program 240. The carrier 250 is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal or a computer readable storage medium.
[0136] By way of example, the software or computer program 240 may be implemented as a computer program product, which is typically carried or stored on a computer-readable medium 250 , in particular a non-volatile medium.
[0137] The computer readable medium may include one or more removable or non-removable storage devices, including (but not limited to): read-only memory (ROM), random access memory (RAM), compact disk (CD), digital versatile disk (DVD), Blu-ray disk, universal serial bus (USB) memory, hard disk drive (HDD) storage device, flash memory, magnetic tape, or any other conventional storage device. Thus, the computer program 240 may be loaded into the operating memory 220 for execution by the processing circuit 210.
[0138] When executed by one or more processors, the one or more flowcharts presented herein may be considered as one or more computer flowcharts. The corresponding device may be defined as a set of functional modules, wherein each step executed by the processor corresponds to a functional module. In this case, the functional module is implemented as a computer program running on the processor.
[0139] Therefore, the computer program resident in the memory may be organized into appropriate functional modules, which are configured to, when executed by the processor, perform at least part of the steps and / or tasks described herein.
[0140] Sample Program Code
[0141] An example implementation of the above algorithm is shown below in Java. Note that this code does not include the implementation of the above segmented object detection which may further improve performance. The implementation wraps the algorithm in a test environment where the following parameters can be specified:
[0142] - Total number of frames – assuming no jitter between incoming frames (i.e., frames incoming every 100 milliseconds).
[0143] -Total number of CMPs. The reputation index of a CMP is randomly generated during runtime as a double precision number between 0 and 1.
[0144] - A coefficient for each incoming frame (ie, every 100 milliseconds) used to calculate the total number of frames that the CMP is able to track, which is the product of the CMP's reputation index and this coefficient.
[0145] - Similarly, a coefficient used to calculate the total number of frames that the CMP is able to detect.
[0146] - Battery cost of a single tracking operation. Similarly, battery cost of a single detection operation.
[0147] The following parameters are used for CMP:
[0148] CMP[0]ID:ID_0Reputation index: 0.6256970200371744
[0149] CMP[1]ID:ID_1Reputation Index: 0.6584154866876781
[0150] CMP[2]ID:ID_2Reputation index: 0.6305296502731899
[0151] CMP[3]ID:ID_3Reputation index: 0.7541284349836521
[0152] CMP[4]ID:ID_4Reputation index: 0.1208759739305121
[0153] CMP[5]ID:ID_5Reputation index: 0.765604648894787
[0154] CMP[6]ID:ID_6Reputation index: 0.9255785744225709
[0155] CMP[7]ID:ID_7Reputation index: 0.06024311960588813
[0156] CMP[8]ID:ID_8Reputation index: 0.08155379830196141
[0157] CMP[9]ID:ID_9Reputation index: 0.8405986416587938
[0158] Using these parameters, the system can perform object detection for 1300 frames before draining the batteries in all 10 CMPs. On the other hand, if a non-distributed client-server model is used (assuming, for example, that the performance metric of the server is as large as the most capable CMP (0.9255 listed above)), the process will end at frame 265 due to the server's battery draining.
[0159]
[0160]
[0161]
[0162]
[0163]
[0164]
[0165]
[0166]
[0167]
[0168]
[0169] The embodiments described above will be understood as several illustrative examples of the present invention. It will be appreciated by those skilled in the art that various modifications, combinations and changes may be made to the embodiments without departing from the scope of the present invention. Specifically, different partial solutions in different embodiments may be combined in other configurations where technically feasible. However, the scope of the present invention is limited by the appended claims.
Claims
1. A method for tracking the position of an object in a series of frames of a video stream, the method comprising: selecting a first node from a set of two or more nodes that can be used for object detection; sending a first current frame from the series of frames to the first node for detecting a first object in the frame; receiving object detection information for the first object from the first node; After receiving the object detection information for the first object from the first node, selecting a second node from a set of two or more nodes available for object detection, and sending a second current frame from the series of frames to the second node for updated detection of the first object; sending each of the two or more frames following the second current frame to a respective tracking node so that the first object is tracked by the respective tracking node, wherein sending each frame to the respective tracking node comprises: selecting the respective tracking node from a set of two or more nodes available for tracking, and wherein each of the two or more frames is sent only to the respective tracking node selected from the set; sending object modeling information derived from the object detection information and indicating the location and / or category of one or more objects to each of the respective tracking nodes; and Tracking information of the first object for a frame sent to the corresponding tracking node is received from each of the corresponding tracking nodes.
2. The method according to claim 1, further comprising: After sending the two or more frames after the second current frame to each corresponding tracking node: receiving updated object detection information for the first object from the second node; A third node is selected from the set of two or more nodes available for object detection, and a third current frame from the series of frames is sent to the third node for further updated detection of the first object.
3. The method according to claim 2, wherein: Sending each of the two or more frames after the second current frame to respective tracking nodes includes sending each frame between the second current frame and the third current frame to tracking nodes.
4. The method according to claim 2, further comprising: After sending the third current frame to the third node: Sending each of the two or more frames following the second current frame to each corresponding tracking node, wherein sending each frame to the corresponding tracking node comprises: selecting the corresponding tracking node from a set of two or more nodes available for tracking; sending updated object modeling information derived from the updated object detection information to each of the respective tracking nodes; and Tracking information of the first object for a frame sent to the corresponding tracking node is received from each of the corresponding tracking nodes.
5. The method according to claim 1, wherein: Sending object modeling information derived from the object detection information and indicating the location and / or category of one or more objects to each of the respective tracking nodes includes sending the object modeling information and the respective frame to each of the respective tracking nodes.
6. The method according to claim 1, wherein: Sending object modeling information derived from the object detection information and indicating the location and / or category of one or more objects to each of the respective tracking nodes includes sending the object modeling information to all nodes in a set of two or more nodes available for tracking.
7. The method according to claim 1, wherein: The method includes: maintaining a single set of nodes that can be used for detection or tracking, wherein the maintenance includes: removing each node from the set when the node is selected for detection or tracking and performs the corresponding detection or tracking; and returning the node to the set when each node is selected for detection and tracking and completes its corresponding detection or tracking.
8. The method according to claim 1, wherein: The method includes: maintaining different sets of nodes that can be used for detection and tracking, respectively, wherein the maintenance includes: removing each node selected for detection or tracking from the corresponding set when the node performs the corresponding detection or tracking; and returning each node selected for detection and tracking to the corresponding set when the node completes its corresponding detection or tracking.
9. The method according to claim 1, wherein: Each selection of a node for object detection is made based on one or more metrics of each node in a set of two or more nodes available for detection, wherein the one or more metrics include or are based on any of the following: The battery status of the corresponding node; a measure of processing resources available at the corresponding nodes; The count of previous object detection tasks completed by the corresponding node; A ratio of object detection tasks completed by a corresponding node to the number of object detection tasks assigned to the corresponding node.
10. The method according to claim 1, wherein: Each selection of a node for object tracking is made based on one or more metrics of each node in a set of two or more nodes available for tracking, wherein the one or more metrics include or are based on any of the following: The battery status of the corresponding node; a measure of processing resources available at the corresponding nodes; a count of previous object tracking tasks completed by the corresponding node; A ratio of object tracking tasks completed by a corresponding node to the number of object tracking tasks assigned to the corresponding node.
11. An arrangement of one or more nodes, each of the one or more nodes comprising a processing circuit and an associated memory, the memory comprising program instructions for execution by the corresponding processing circuit, the program instructions being configured to track the position of an object in a series of frames of a video stream by: selecting a first node from a set of two or more nodes that can be used for object detection; sending a first current frame from the series of frames to the first node for detecting a first object in the frame; receiving object detection information for the first object from the first node; After receiving the object detection information for the first object from the first node, selecting a second node from a set of two or more nodes available for object detection, and sending a second current frame from the series of frames to the second node for updated detection of the first object; sending each of the two or more frames following the second current frame to a respective tracking node so that the first object is tracked by the respective tracking node, wherein sending each frame to the respective tracking node comprises: selecting the respective tracking node from a set of two or more nodes available for tracking, and wherein each of the two or more frames is sent only to the respective tracking node selected from the set; sending object modeling information derived from the object detection information and indicating the location and / or category of one or more objects to each of the respective tracking nodes; and Tracking information of the first object for a frame sent to the corresponding tracking node is received from each of the corresponding tracking nodes.
12. The arrangement device according to claim 11, wherein: The program instructions are further configured to, after sending the two or more frames after the second current frame to each corresponding tracking node, cause the node to: receiving updated object detection information for the first object from the second node; A third node is selected from the set of two or more nodes available for object detection, and a third current frame from the series of frames is sent to the third node for further updated detection of the first object.
13. The arrangement device according to claim 12, wherein: The program instructions are configured such that sending each of the two or more frames following the second current frame to respective tracking nodes includes sending each frame between the second current frame and the third current frame to tracking nodes.
14. The arrangement device according to claim 12, wherein: The program instructions are further configured to: after sending the third current frame to the third node, cause the node to: Sending each of the two or more frames following the second current frame to each corresponding tracking node, wherein sending each frame to the corresponding tracking node comprises: selecting the corresponding tracking node from a set of two or more nodes available for tracking; sending updated object modeling information derived from the updated object detection information to each of the respective tracking nodes; and Tracking information of the first object for a frame sent to the corresponding tracking node is received from each of the corresponding tracking nodes.
15. The arrangement device according to claim 11, wherein: The program instructions are configured such that sending object modeling information derived from the object detection information and indicating the location and / or category of one or more objects to each of the respective tracking nodes includes sending the object modeling information and the respective frame to each of the respective tracking nodes.
16. The arrangement device according to claim 11, wherein: The program instructions are configured such that sending object modeling information derived from the object detection information and indicating the location and / or category of one or more objects to each of the respective tracking nodes includes sending the object modeling information to all nodes in a set of two or more nodes available for tracking.
17. The arrangement device according to claim 11, wherein: The program instructions are also configured to cause the node to maintain a single set of nodes that can be used for detection or tracking, wherein the maintenance includes: removing the node from the set when each node selected for detection or tracking performs the corresponding detection or tracking; and returning the node to the set when each node selected for detection and tracking completes its corresponding detection or tracking.
18. The arrangement device according to claim 11, wherein: The program instructions are configured to cause the nodes to respectively maintain different sets of nodes that can be used for detection and tracking, wherein the maintenance includes: removing each node selected for detection or tracking from the corresponding set when the node performs the corresponding detection or tracking; and returning each node selected for detection and tracking to the corresponding set when the node completes its corresponding detection or tracking.
19. The arrangement device according to claim 11, wherein: The program instructions are configured such that each selection of a node for object detection is based on one or more metrics of each node in a set of two or more nodes available for detection, wherein the one or more metrics include or are based on any of the following: The battery status of the corresponding node; a measure of processing resources available at the corresponding nodes; The count of previous object detection tasks completed by the corresponding node; A ratio of object detection tasks completed by a corresponding node to the number of object detection tasks assigned to the corresponding node.
20. The arrangement device according to claim 11, wherein: The program instructions are configured such that each selection of a node for object tracking is based on one or more metrics of each node in a set of two or more nodes available for tracking, wherein the one or more metrics include or are based on any of the following: The battery status of the corresponding node; a measure of processing resources available at the corresponding nodes; a count of previous object tracking tasks completed by the corresponding node; A ratio of object tracking tasks completed by a corresponding node to the number of object tracking tasks assigned to the corresponding node.
21. A device for arranging one or more nodes, wherein the one or more nodes are configured to: selecting a first node from a set of two or more nodes that can be used for object detection; sending a first current frame from a series of frames to the first node for detecting a first object in the frame; receiving object detection information for the first object from the first node; After receiving the object detection information for the first object from the first node, selecting a second node from a set of two or more nodes available for object detection, and sending a second current frame from the series of frames to the second node for updated detection of the first object; sending each of the two or more frames following the second current frame to a respective tracking node so that the first object is tracked by the respective tracking node, wherein sending each frame to the respective tracking node comprises: selecting the respective tracking node from a set of two or more nodes available for tracking, and wherein each of the two or more frames is sent only to the respective tracking node selected from the set; sending object modeling information derived from the object detection information and indicating the location and / or category of one or more objects to each of the respective tracking nodes; and Tracking information of the first object for a frame sent to the corresponding tracking node is received from each of the corresponding tracking nodes.
22. The arrangement according to claim 21, wherein: The one or more nodes are configured to perform a method according to any one of claims 2-10.
23. A computer readable medium having stored thereon computer program instructions for tracking the position of an object in a series of frames of a video stream, the computer program instructions comprising instructions for: selecting a first node from a set of two or more nodes that can be used for object detection; sending a first current frame from the series of frames to the first node for detecting a first object in the frame; receiving object detection information for the first object from the first node; After receiving the object detection information for the first object from the first node, selecting a second node from a set of two or more nodes available for object detection, and sending a second current frame from the series of frames to the second node for updated detection of the first object; sending each of the two or more frames following the second current frame to a respective tracking node so that the first object is tracked by the respective tracking node, wherein sending each frame to the respective tracking node comprises: selecting the respective tracking node from a set of two or more nodes available for tracking, and wherein each of the two or more frames is sent only to the respective tracking node selected from the set; sending object modeling information derived from the object detection information and indicating the location and / or category of one or more objects to each of the respective tracking nodes; and Tracking information of the first object for a frame sent to the corresponding tracking node is received from each of the corresponding tracking nodes.
