Event camera multi-target tracking method, device and equipment based on multi-view representation
By employing multi-view representation and multi-plane projection methods, the observation difficulties and trajectory overlap issues encountered by event cameras in multi-target tracking were resolved, achieving high-precision multi-target tracking results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2026-03-17
AI Technical Summary
Existing technologies struggle to effectively observe and track small targets, high-speed motion, and high-dynamic environments in multi-target tracking based on frame cameras. Furthermore, the data encoding method of event cameras makes traditional algorithms prone to errors when trajectories overlap or intersect.
The multi-view representation method is adopted. The motion process of multiple targets in space is captured by an event camera. Event flow segmentation and multi-plane projection are performed. Combined with Gaussian pyramid processing, two-dimensional trajectory regions are extracted and trajectory segments of the same trajectory are merged to form multiple trajectories, and finally, multi-target tracking is completed.
It improves the accuracy and precision of multi-target tracking, solves the problem of inconsistent time spans for targets with different moving speeds, avoids detection errors when trajectories overlap or intersect, and achieves efficient target tracking results.
Smart Images
Figure CN119131082B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus and device for multi-target tracking of event cameras based on multi-view representation. Background Technology
[0002] Multi-object tracking is a challenging task in the field of computer vision, with a wide range of applications.
[0003] In the field of frame camera-based tracking, many high-precision methods have emerged for multi-target tracking tasks. However, frame camera-based observation is difficult in situations involving small targets, high-speed motion, and high dynamics, making it impossible to perform subsequent tasks.
[0004] Therefore, the field of event cameras has emerged. The advantage of event cameras is that they can reduce the difficulty of observing small targets with weak textures in extreme environments. However, their data encoding method is different from that of traditional frame cameras, which makes it necessary to redesign the corresponding algorithms according to the characteristics of event camera data.
[0005] In existing technologies, event data processing methods are generally divided into two types: event-by-event and pseudo-event frames. The former usually uses clustering algorithms to separate different target clusters according to parameters such as distance. Although it makes full use of the spatiotemporal information of the data, the clustering parameters are difficult to confirm when the time span of targets with different moving speeds is inconsistent. The latter transforms the data stream into a two-dimensional image. This method directly compresses some dimensional event information, and tracking errors are easily detected when different trajectories partially overlap or intersect. Summary of the Invention
[0006] Therefore, it is necessary to provide a method, apparatus, and device for multi-target tracking of event cameras based on multi-view representation to address the above-mentioned technical problems. This method can improve the accuracy of multi-target tracking, display the spatiotemporal information of the event stream in a more intuitive way, and is also compatible with various image segmentation algorithms.
[0007] Multi-view representation-based event camera multi-target tracking methods include:
[0008] An event camera is used to capture the motion of multiple targets in space, resulting in an event stream; the event stream is then segmented to obtain event blocks.
[0009] For each event block, after time realignment mapping, multi-plane projection is performed to obtain a multi-view representation of each event block;
[0010] For each view, perform segmentation and iterate through all non-zero pixels to obtain the two-dimensional trajectory region of each view, which includes multiple trajectory segments.
[0011] For multiple views, merge all trajectory segments of the same trajectory to obtain multiple trajectory domains for each event block; for each event block, merge all trajectory domains of the same trajectory to obtain multiple trajectory segments for each event block; for all event blocks, merge all trajectory segments of the same trajectory to obtain multiple trajectories for the event flow.
[0012] Multi-target tracking is achieved based on multiple trajectories in the event stream.
[0013] In one embodiment, for each event block, after time realignment mapping, multi-plane projection is performed to obtain a multi-view representation of each event block, including:
[0014] Choose three orthogonal planes in a three-dimensional Cartesian coordinate system as the multi-view projection planes;
[0015] For each event block, perform time realignment mapping to obtain realigned event blocks;
[0016] The realigned event blocks are projected onto a multi-view projection plane and passed through a multi-scale Gaussian pyramid to obtain a multi-view representation of each event block.
[0017] In one embodiment, time realignment mapping is performed for each event block, including:
[0018]
[0019] In the formula, T′ is the set of pseudo timestamps, T is the set of original timestamps for the event block, and t i ' is a constituent element of T', t i Let g be a component of T, and g(·) be a relabeling function.
[0020] In one embodiment, projecting the realign event block onto a multi-view projection plane includes: projecting the realign event block onto the XOY view projection plane, the YOT view projection plane, and the TOX view projection plane.
[0021] The process of projecting the realign event block onto the YOT view projection plane includes:
[0022]
[0023] In the formula, e i For the i-th event, (t) i ,x i ,y i ,p i Let be the timestamp, x-coordinate, y-coordinate, and polarity of the i-th event, respectively; E be the event block; and δ(·) be the Dirac function.
[0024] In one embodiment, for each view, segmentation is performed, and all non-zero pixels are traversed to obtain a two-dimensional trajectory region for each view. The trajectory region includes multiple trajectory segments, including:
[0025] The gradient magnitude of each view is calculated using a weighted gradient operator.
[0026] Sort all non-zero pixels in each view in descending order of gradient magnitude to form a seed list, and use the non-zero pixel at the head of the seed list as the initial seed point.
[0027] If a non-zero pixel exists in the specified neighborhood of the initial seed point and has not been visited, add the non-zero pixel to the end of the region list and the seed list, and mark the non-zero pixel as visited; otherwise, continue searching until the specified neighborhood of the initial seed point has been traversed.
[0028] Using the next non-zero pixel of the initial seed point in the seed list as the next seed point, search the specified neighborhood of the next seed point until the seed list is traversed to obtain all trajectory extraction results after traversing all non-zero pixels of the projection surface. Based on the region list, delete regions with trajectory areas smaller than the preset value to obtain the two-dimensional trajectory region of each view. The trajectory region includes multiple trajectory segments.
[0029] In one embodiment, for multiple views, all trajectory segments of the same trajectory are merged to obtain multiple trajectory domains for each event block, including:
[0030] For two views containing a time axis in a multi-view dataset, sort the trajectory regions in descending order of area and calculate the planar region parameters for each trajectory region.
[0031] The trajectory region with the largest area is taken as the first trajectory region. All trajectory regions that meet the first preset condition are merged into the first trajectory region to obtain the second trajectory region.
[0032] Calculate the planar region parameters of the second trajectory region, and search and merge again until merging is no longer possible;
[0033] Traverse all unmerged trajectory regions, perform calculations, searches, and merges to merge all trajectory segments of the same trajectory in two views containing the time axis;
[0034] All trajectory segments of the same trajectory in the two views containing the time axis, as well as the trajectory regions of all views not containing the time axis, are back-projected into the 3D event space to obtain multiple trajectory domains for each event block.
[0035] In one embodiment, for each event block, all trajectory domains of the same trajectory are merged to obtain multiple trajectory segments for each event block, including:
[0036] For each event block, calculate the XYT spatial parameters for all trajectory domains;
[0037] The trajectory domain with the most points is selected as the first trajectory domain, and a search is performed to merge other trajectory domains that meet the second preset condition into the first trajectory domain to obtain the second trajectory domain, and the XYT space parameters of the second trajectory domain are updated.
[0038] Repeat the search, merge, and update parameters until merging is no longer possible. Trajectory domains that have not been merged are traversed to obtain multiple trajectory segments for each event block.
[0039] In one embodiment, for all event blocks, all trajectory segments of the same trajectory are merged to obtain multiple trajectories of the event flow, including:
[0040] Calculate the XYT spatial parameters of all trajectory segments in the current event block and the n adjacent event blocks respectively;
[0041] If there is a trajectory segment in the first n event blocks that satisfies the third preset condition with the trajectory in the current event block, then all trajectory segments of the same trajectory in the first n event blocks and the current event block are merged; if not, then continue to search for trajectory segments whose intersection point is located in the spatiotemporal region between the two trajectory segment extension points, and merge them; until all event blocks are traversed, and multiple trajectories of the event flow are obtained.
[0042] A multi-view representation-based event camera multi-target tracking device includes:
[0043] The acquisition module is used to capture the motion process of multiple targets in space using an event camera to obtain an event stream; the event stream is then segmented to obtain event blocks.
[0044] The projection module is used to perform multi-plane projection on each event block after time realignment mapping to obtain a multi-view representation of each event block;
[0045] The segmentation module is used to segment each view and traverse all non-zero pixels to obtain the two-dimensional trajectory region of each view. The trajectory region includes multiple trajectory segments.
[0046] The merging module is used to merge all trajectory segments of the same trajectory in multiple views to obtain multiple trajectory fields for each event block; for each event block, it merges all trajectory fields of the same trajectory to obtain multiple trajectory segments for each event block; and for all event blocks, it merges all trajectory segments of the same trajectory to obtain multiple trajectories of the event flow.
[0047] The tracking module is used to perform multi-target tracking based on multiple trajectories in the event stream.
[0048] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:
[0049] An event camera is used to capture the motion of multiple targets in space, resulting in an event stream; the event stream is then segmented to obtain event blocks.
[0050] For each event block, after time realignment mapping, multi-plane projection is performed to obtain a multi-view representation of each event block;
[0051] For each view, perform segmentation and iterate through all non-zero pixels to obtain the two-dimensional trajectory region of each view, which includes multiple trajectory segments.
[0052] For multiple views, merge all trajectory segments of the same trajectory to obtain multiple trajectory domains for each event block; for each event block, merge all trajectory domains of the same trajectory to obtain multiple trajectory segments for each event block; for all event blocks, merge all trajectory segments of the same trajectory to obtain multiple trajectories for the event flow.
[0053] Multi-target tracking is achieved based on multiple trajectories in the event stream.
[0054] The aforementioned event camera multi-target tracking method, apparatus, and device based on multi-view representation utilize the temporal and spatial information contained in the event for three-plane representation. Combining the high temporal resolution of the event camera, the target tracking problem is transformed into a trajectory segmentation problem. An event camera target tracking structure that can apply traditional image frame algorithms is constructed. Accurate target tracking results are obtained by utilizing the complementary trajectory information of different planes. The measurement method is precise and efficient, and the process is simple and convenient. It can be widely used for related target tracking, target segmentation, and other tasks. Attached Figure Description
[0055] Figure 1 This is an application scenario diagram of an event camera multi-target tracking method based on multi-view representation in one embodiment;
[0056] Figure 2 This is a flowchart illustrating an event camera multi-target tracking method based on multi-view representation in one embodiment;
[0057] Figure 3 This is a schematic diagram based on a three-plane representation in one embodiment;
[0058] Figure 4 This is a structural block diagram of an event camera multi-target tracking device based on multi-view representation in one embodiment;
[0059] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0061] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.
[0062] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. In the description of this application, "multiple sets" means at least two sets, such as two sets, three sets, etc., unless otherwise explicitly specified.
[0063] In this application, unless otherwise expressly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two elements or the interaction between two elements, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.
[0064] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.
[0065] The event camera multi-target tracking method based on multi-view representation provided in this application can be applied to, for example... Figure 1 The application scenario diagram shows that terminal 102 communicates with server 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 104 can be a server corresponding to the backend of various portal websites or work systems.
[0066] This application provides a multi-target tracking method for event cameras based on multi-view representation, such as... Figure 2 The flowchart shown illustrates, in one embodiment, the method is applied to Figure 1 Taking the terminal in the example, the explanation includes:
[0067] Step 202: Use an event camera to capture the motion process of multiple targets in space to obtain an event stream; segment the event stream to obtain event blocks.
[0068] In this step, segmentation includes: segmenting according to a fixed time interval or a fixed number of events.
[0069] Step 204: For each event block, after time realignment mapping, perform multi-plane projection to obtain a multi-view representation of each event block.
[0070] Specifically:
[0071] Three orthogonal planes XOY, YOT, and TOX in a three-dimensional Cartesian coordinate system are selected as multi-view projection planes. The multi-view projection planes include: XOY view projection plane, YOT view projection plane and TOX view projection plane.
[0072] For all events in each event block, perform time realignment mapping to obtain realigned event blocks, thereby achieving a more temporally dense event distribution;
[0073] The realigned event blocks are projected onto the multi-view projection plane (the realigned event blocks are projected onto the multi-view projection plane in the same way as ordinary event frames without considering polarity), and then processed through a multi-scale Gaussian pyramid (multi-scale fusion processing is performed using a pyramid with Gaussian sampling) to obtain a multi-view representation of each event block, which is a projected view on three planes.
[0074] For each event block, a time realignment mapping is performed based on the time correspondence function:
[0075]
[0076] In the formula, T′ is the set of pseudo timestamps, T is the set of original timestamps for the event block, and t i ' is a component of T', i.e., a pseudo-timestamp, t i The constituent elements of T are the original timestamps, t i ′ and t i One-to-one correspondence, g(·) is a relabeling function, belonging to the time correspondence function;
[0077] Taking the projection of the realign event block onto the YOT view projection plane as an example:
[0078]
[0079] In the formula, e i For the i-th event, (t) i ,x i ,y i ,p i ) are the timestamp, x-coordinate, y-coordinate, and polarity of the i-th event, respectively, E is the event block, and δ(·) is the Dirac delta function.
[0080] In this step, the three planes are represented as follows: Figure 3 As shown, after time realignment mapping, the event block is directly projected onto the XOY, YOT, and TOX planes, and then multi-scale fusion is performed through a Gaussian pyramid to output a multi-view representation.
[0081] Event blocks, after time realignment mapping, can solve the problem of difficulty in determining clustering parameters when targets with different motion speeds have inconsistent time spans in existing technologies.
[0082] Step 206: For each view, perform segmentation processing and traverse all non-zero pixels to obtain the two-dimensional trajectory region of each view. The trajectory region includes multiple trajectory segments.
[0083] Specifically:
[0084] The gradient magnitude for each view is calculated using the Gaussian weighted gradient operator.
[0085] Sort all non-zero pixels in each view in descending order of gradient magnitude to form a seed list, and use the non-zero pixel at the head of the seed list (i.e. the one with the largest gradient magnitude) as the initial seed point.
[0086] If a non-zero pixel exists in the specified neighborhood of the initial seed point and has not been visited (meaning it has not been searched), then the non-zero pixel is added to the end of the region list (the region list is the set of regions where the trajectory is located) and the seed list (the seed list is the set of seed points), and the non-zero pixel is marked as visited; otherwise, the search continues until the specified neighborhood of the initial seed point is traversed.
[0087] Using the next non-zero pixel of the initial seed point in the seed list as the next seed point, search the specified neighborhood of the next seed point until the seed list is traversed to obtain all trajectory extraction results after traversing all non-zero pixels of the projection surface. Based on the region list, delete regions whose trajectory area is less than a preset value (the preset value can be determined according to existing technology) to obtain the two-dimensional trajectory region of each view. The trajectory region includes multiple trajectory segments.
[0088] In this step, the segmentation process is just one example; of course, other segmentation methods in the prior art can also be used.
[0089] Step 208: For multiple views, merge all trajectory segments of the same trajectory to obtain multiple trajectory domains for each event block; for each event block, merge all trajectory domains of the same trajectory to obtain multiple trajectory segments for each event block; for all event blocks, merge all trajectory segments of the same trajectory to obtain multiple trajectories of the event flow.
[0090] Specifically:
[0091] For multiple views, all trajectory segments of the same trajectory are merged to obtain multiple trajectory domains for each event block. This includes: for two views (YOT and TOX) containing a time axis in the multiple views, the trajectory regions are sorted in descending order of area, and the planar region parameters of each trajectory region are calculated; the trajectory region with the largest area is taken as the first trajectory region, and a search is performed based on the planar region parameters to merge all trajectory regions that meet the first preset condition into the first trajectory region to obtain the second trajectory region; the planar region parameters of the second trajectory region are calculated, and the search and merging are repeated until merging is no longer possible; all trajectory regions that have not been merged are traversed, and calculation, search, and merging are performed to merge all trajectory segments of the same trajectory in the two views containing the time axis; all trajectory segments of the same trajectory in the two views containing the time axis and the trajectory regions of all views not containing the time axis are back-projected into the three-dimensional event space to obtain a trajectory set composed of event points; each trajectory in the three planar trajectory sets is traversed, and the intersection of each pair of trajectories is saved as a new trajectory segment to fuse the results of different views and obtain multiple trajectory domains for each event block. Taking the planar region parameters, including the centroid, direction vector, and region width, as an example, the trajectory regions waiting to be merged into other regions are designated as merging regions, and the trajectory regions other than the merging regions are designated as regions to be merged. The first preset condition is that the distance between the centroid of the region to be merged and the straight line containing the direction vector of the merging region is less than or equal to the region width of the merging region. Specifically: calculate the centroid, direction vector, and region width of each trajectory region; take the trajectory region with the largest area as the first trajectory region, and form a straight line with the centroid of the first trajectory region as the starting point and the direction vector as the direction, and perform a search to merge all trajectory regions whose centroid and the straight line distance is not greater than the region width of the first trajectory region into the first trajectory region to obtain the second trajectory region; calculate the centroid, direction vector, and region width of the second trajectory region, and search and merge again until no more can be merged.
[0092] For each event block, merge all trajectory domains of the same trajectory to obtain multiple trajectory segments for each event block, including: for each event block, calculate the XYT spatial parameters of all trajectory domains; take the trajectory domain with the most points as the first trajectory domain, and search according to the XYT spatial parameters to merge other trajectory domains that meet the second preset condition into the first trajectory domain to obtain the second trajectory domain, and calculate the XYT spatial parameters of the second trajectory domain; re-search, merge, and update parameters until merging is no longer possible, traverse all trajectory domains that have not been merged to obtain multiple trajectory segments for each event block. Taking the XYT space parameters, including the centroid, direction vector, and region width of the trajectory domain, as an example, the second preset condition is that the distance between two trajectory domains is less than the region width of any trajectory domain. Specifically: calculate the centroid c, direction vector d, and region width w of all trajectory domains within the current event block. Take the trajectory domain with the most points as the initial region (i.e., the first trajectory domain). With the centroid as the starting point and the direction vector as the direction, form a straight line. Merge all trajectory domains whose distance from the current trajectory domain is not greater than the width of the current trajectory domain, until all trajectory domains have been merged. Merge trajectory domains whose direction difference is less than a threshold (this threshold can be preset according to existing technology) to obtain the complete event trajectory of the current event block.
[0093] For all event blocks, merge all trajectory segments of the same trajectory to obtain multiple trajectories of the event flow, including: calculating the XYT spatial parameters of all trajectory segments in the current event block and the adjacent n event blocks respectively; if there is a trajectory segment in the preceding n event blocks that satisfies the third preset condition with the trajectory segment in the current event block (the third preset condition is: the angle between the trajectory segments in the current event block and the adjacent n event blocks is less than a preset range), then merge all trajectory segments of the same trajectory in the preceding n event blocks and the current event block; if not, continue to search for trajectory segments that satisfy the condition that the intersection of the extension lines of the trajectory segments is located in the spatiotemporal region between the extension points of the two trajectory segments, and merge them; until all event blocks are traversed to obtain multiple trajectories of the event flow. Specifically: Calculate the centroid and direction vector of all trajectories (Current) appearing in the current event block; calculate the centroid and direction vector of all trajectories (Previous) appearing in the preceding n event blocks; associate the trajectories in Current with those in Previous, forming a straight line with the centroid as the starting point and the direction vector as the direction. If there is a trajectory with minimal direction change and not exceeding a certain value (this value can be preset according to existing technology), then merge them. If there are still trajectories in Current without a corresponding one, then jointly judge with the remaining trajectories in Previous to see if there is an intersection point of the trajectory extensions located within the spatiotemporal region of the two trajectory extension points. If so, merge; otherwise, save the current trajectory as a new trajectory. It should be noted that the search for the preceding n event blocks here uses the divide-and-conquer approach. If the same trajectory can be found in the preceding n event blocks, it means that the current trajectory belongs to a previously appearing trajectory; if not, it means that the trajectory is newly appearing. Therefore, searching and merging in the current and previous event blocks, processing the current event block each time, allows the current and previous trajectories to be associated.
[0094] In this step, multiple views representing different information are fused, which solves the problem of easy detection of tracking errors when different trajectories partially overlap or intersect in the prior art.
[0095] Step 210: Complete multi-target tracking based on multiple trajectories of the event stream.
[0096] In this step, the multiple trajectories of the event stream correspond to the motion processes of multiple targets in space, thereby enabling multi-target tracking.
[0097] The aforementioned event camera multi-target tracking method based on multi-view representation utilizes the temporal and spatial information contained in the event for three-plane representation. Combining the high temporal resolution of the event camera, the target tracking problem is transformed into a trajectory segmentation problem. It is compatible with various image segmentation algorithms and constructs an event camera target tracking structure that can apply traditional image frame algorithms. It obtains accurate target tracking results by utilizing the complementary trajectory information of different planes. The measurement method is precise and efficient, and the process is simple and convenient. It can be widely used for related target tracking, target segmentation and other tasks.
[0098] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0099] This application also provides an event camera multi-target tracking device based on multi-view representation, such as... Figure 4 As shown, in one embodiment, it includes: an acquisition module 402, a projection module 404, a segmentation module 406, a merging module 408, and a tracking module 410, wherein:
[0100] The acquisition module 402 is used to capture the motion process of multiple targets in space using an event camera to obtain an event stream; and to segment the event stream to obtain event blocks.
[0101] The projection module 404 is used to perform multi-plane projection on each event block after time realignment mapping to obtain a multi-view representation of each event block;
[0102] The segmentation module 406 is used to segment each view and traverse all non-zero pixels to obtain the two-dimensional trajectory region of each view. The trajectory region includes multiple trajectory segments.
[0103] The merging module 408 is used to merge all trajectory segments of the same trajectory in multiple views to obtain multiple trajectory fields for each event block; for each event block, it merges all trajectory fields of the same trajectory to obtain multiple trajectory segments for each event block; and for all event blocks, it merges all trajectory segments of the same trajectory to obtain multiple trajectories of the event flow.
[0104] The tracking module 410 is used to complete multi-target tracking based on multiple trajectories of the event stream.
[0105] Specific limitations regarding the event camera multi-target tracking device based on multi-view representation can be found in the limitations of the event camera multi-target tracking method based on multi-view representation above, and will not be repeated here. Each module in the above device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.
[0106] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a multi-target tracking method based on multi-view representation of an event camera. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.
[0107] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0108] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.
[0109] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.
[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0111] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0112] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for multi-target tracking of an event camera based on a multi-view representation, characterized in that, The application relates to a method for tracking multiple targets in a space, comprising the following steps: a space multiple target motion process is photographed by using an event camera to obtain an event stream; the event stream is segmented to obtain an event block; after time realignment mapping, multi-plane projection is performed on each event block to obtain a multi-view representation of each event block; segmentation processing is performed on each view, and all non-zero pixel points are traversed to obtain a two-dimensional trajectory area of each view, the trajectory area comprising multiple trajectory segments; for the multi-view, all trajectory segments of the same trajectory are merged to obtain multiple trajectory domains of each event block; for each event block, all trajectory domains of the same trajectory are merged to obtain multiple trajectory segments of each event block; for all event blocks, all trajectory segments of the same trajectory are merged to obtain multiple trajectories of the event stream; multiple target tracking is completed according to the multiple trajectories of the event stream; segmentation processing is performed on each view, and all non-zero pixel points are traversed to obtain a two-dimensional trajectory area of each view, the trajectory area comprising multiple trajectory segments, comprising the following steps: a weighted gradient operator is used to calculate the gradient amplitude of each view; all non-zero pixel points of each view are sorted in descending order of gradient amplitude to obtain a seed list, and a non-zero pixel point at the head of the seed list is taken as an initial seed point; a specified neighborhood of the initial seed point is searched, if there is a non-zero pixel point and it has not been visited, the non-zero pixel point is added to the end of the region list and the seed list, and the non-zero pixel point is set as having been visited; otherwise, the search is continued until the specified neighborhood of the initial seed point is traversed; the next non-zero pixel point of the initial seed point in the seed list is taken as a next seed point, and a specified neighborhood of the next seed point is searched until the seed list is traversed, so that all trajectory extraction results of the projection plane are obtained, and according to the region list, regions with an area smaller than a preset value are deleted to obtain a two-dimensional trajectory area of each view, the trajectory area comprising multiple trajectory segments; for the multi-view, all trajectory segments of the same trajectory are merged to obtain multiple trajectory domains of each event block, comprising the following steps: for two views containing a time axis in the multi-view, the trajectory areas are arranged in descending order of area, and the plane area parameters of each trajectory area are calculated; the largest trajectory area is taken as a first trajectory area, and all trajectory areas meeting a first preset condition are merged into the first trajectory area to obtain a second trajectory area; the plane area parameters of the second trajectory area are calculated, and the search and the merge are restarted until the merge is impossible; all trajectory areas not subjected to the merge are traversed to perform calculation, search and merge, so that all trajectory segments of the same trajectory in the two views containing the time axis are merged; all trajectory segments of the same trajectory in the two views containing the time axis and the trajectory areas of all views not containing the time axis are back-projected into a three-dimensional event space to obtain multiple trajectory domains of each event block.
2. The multi-view representation based event camera multi-target tracking method of claim 1, wherein, after time realignment mapping, multi-plane projection is performed on each event block to obtain a multi-view representation of each event block, comprising the following steps: three orthogonal planes in a three-dimensional Cartesian rectangular coordinate system are selected as multi-view projection planes; time realignment mapping is performed on each event block to obtain a realignment event block; Projecting the realigned event blocks to multi-view projection planes and through a multi-scale Gaussian pyramid to obtain multi-view representations of each event block.
3. The multi-view representation based event camera multi-target tracking method of claim 2, wherein, For each event block, performing time realignment mapping, including: wherein is a set of pseudo timestamps, is a set of original timestamps of the event block, is a set of original timestamps of the event block, is a constituent element of is a constituent element of is a constituent element of is a relabeling function.
4. The multi-view representation based event camera multi-target tracking method of claim 3, wherein, Projecting the realigned event blocks to multi-view projection planes, including projecting the realigned event blocks to XOY, YOT and TOX view projection planes; Projecting the realigned event blocks to the YOT view projection plane includes: wherein is the i event, is the i timestamp, x-coordinate, y-coordinate, polarity, respectively, of the event block, is the Dirac function.
5. The multi-view representation based event camera multi-target tracking method according to any one of claims 1 to 4, characterized in that, For each event block, merging all track domains of the same track to obtain multiple track segments of each event block, including: For each event block, calculating the XYT space parameters of all track domains; Taking the track domain with the most points as the first track domain and performing searching to merge other track domains meeting a second preset condition to the first track domain to obtain a second track domain and update the XYT space parameters of the second track domain; Re-searching, merging and updating parameters until no merging is possible, and traversing all track domains that have not been merged to obtain multiple track segments of each event block.
6. The multi-view representation based event camera multi-target tracking method according to any one of claims 1 to 4, characterized in that, For all event blocks, merging all track segments of the same track to obtain multiple tracks of the event stream, including: The XYT space parameters of all trajectory segments in the current event block and the adjacent previous event block are calculated respectively. The XYT space parameters of all trajectory segments in the current event block and the adjacent previous event block are calculated respectively. If before If a trajectory segment exists within an event block that satisfies the third preset condition along with the trajectory in the current event block, then before merging... If an event block is not satisfied with the same trajectory as the current event block, the search continues to find trajectory segments whose intersection point is located within the spatiotemporal region of the two trajectory segment extension points, and these segments are merged; until all event blocks are traversed, multiple trajectories of the event flow are obtained.
7. Apparatus for multi-target tracking of an event camera based on a multi-view representation, characterized in that, The method for multi-target tracking of an event camera based on multi-view representations includes: An acquisition module is configured to capture a motion process of a spatial multi-target by using an event camera to obtain an event stream, and segment the event stream to obtain event blocks; A projection module is configured to, for each event block, perform multi-plane projection after time realignment mapping to obtain a multi-view representation of each event block; A segmentation module is configured to, for each view, perform segmentation processing and traverse all non-zero pixel points to obtain a two-dimensional track region of each view, the track region including multiple track segments; A merging module is configured to, for multiple views, merge all track segments of the same track to obtain multiple track domains of each event block, for each event block, merge all track domains of the same track to obtain multiple track segments of each event block, and for all event blocks, merge all track segments of the same track to obtain multiple tracks of the event stream; A tracking module is configured to complete multi-target tracking according to the multiple tracks of the event stream.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.