Event camera multi-target tracking method, device and equipment based on multi-view representation

Abstract

The invention belongs to the technical field of data processing, and relates to an event camera multi-target tracking method, device and equipment based on multi-view representation. The method comprises the following steps: shooting multiple targets by adopting an event camera to obtain event blocks; performing multi-plane projection on each event block after time realignment mapping to obtain a multi-view representation of each event block; each view is segmented, all non-zero pixel points are traversed, a two-dimensional track area of each view is obtained, and the track area comprises a plurality of track segments; for multiple views, combining all track segments of the same track to obtain a plurality of track domains of each event block; for each event block, combining all trajectory domains of the same trajectory to obtain a plurality of trajectory segments of each event block; for all the event blocks, combining all trajectory segments of the same trajectory to obtain a plurality of trajectories of the event stream; and completing multi-target tracking according to the plurality of tracks of the event stream. The method can improve the multi-target tracking precision, and is more visual.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and equipment for event camera multi-target tracking based on multi-view representation. Background Art

[0002] Multi-object tracking is a challenging task in computer vision with a wide range of applications.

[0003] In the field of frame-based cameras, many high-precision methods have emerged for multi-target tracking tasks. However, in the case of small targets, high-speed motion, and high dynamics, frame-based camera observations are difficult, making subsequent tasks impossible.

[0004] Therefore, the field of event cameras came into being. The advantage of event cameras is that they can reduce the difficulty of observing small targets with weak textures in extreme environments, but their data encoding method is different from that of traditional frame cameras, which requires the corresponding algorithms to be redesigned according to the data characteristics of event cameras.

[0005] In the prior art, the means of processing event data are usually divided into two types: event-by-event and pseudo-event frame. The former usually uses a clustering algorithm to separate different target clusters according to parameters such as distance. Although it makes full use of the spatiotemporal information of the data, when the spans of targets with different moving speeds in the time dimension are inconsistent, the clustering parameters are difficult to confirm. The latter transforms the data stream into a two-dimensional image. This method directly compresses the event information of some dimensions, and it is easy to detect tracking errors when different trajectories partially overlap or intersect. Summary of the invention

[0006] Based on this, it is necessary to provide an event camera multi-target tracking method, device and equipment based on multi-view representation to address the above-mentioned technical problems, which can improve the multi-target tracking accuracy and display the spatiotemporal information of the event stream in a more intuitive way. In addition, it can also adapt to a variety of image segmentation algorithms.

[0007] The event camera multi-target tracking method based on multi-view representation includes:

[0008] The event camera is used to shoot the motion process of multiple targets in space to obtain an event stream; the event stream is segmented to obtain event blocks;

[0009] For each event block, after time realignment mapping, multi-plane projection is performed to obtain a multi-view representation of each event block;

[0010] For each view, segmentation processing is performed, and all non-zero pixels are traversed to obtain a two-dimensional trajectory area for each view, and the trajectory area includes multiple trajectory segments;

[0011] For multiple views, all the track segments of the same track are merged to obtain multiple track domains for each event block; for each event block, all the track domains of the same track are merged to obtain multiple track segments for each event block; for all event blocks, all the track segments of the same track are merged to obtain multiple tracks of the event stream;

[0012] Based on multiple trajectories of the event stream, multi-target tracking is completed.

[0013] In one embodiment, for each event block, after time realignment mapping, multi-plane projection is performed to obtain a multi-view representation of each event block, including:

[0014] Select three orthogonal planes in a three-dimensional Cartesian rectangular coordinate system as multi-view projection planes;

[0015] For each event block, time realignment mapping is performed to obtain a realigned event block;

[0016] The realigned event blocks are projected onto the multi-view projection plane and passed through a multi-scale Gaussian pyramid to obtain a multi-view representation of each event block.

[0017] In one embodiment, time realignment mapping is performed on each event block, including:

[0018]

[0019] Where T′ is the pseudo timestamp set, T is the original timestamp set of the event block, and t i ' is the constituent element of T', t i are the components of T, and g(·) is the relabeling function.

[0020] In one embodiment, projecting the realignment event block onto the multi-view projection plane includes: projecting the realignment event block onto the XOY view projection plane, the YOT view projection plane, and the TOX view projection plane;

[0021] Among them, projecting the realignment event block onto the YOT view projection plane includes:

[0022]

[0023] In the formula, e i is the ith event, (t i ,x i ,y i ,p i ) are the timestamp, abscissa, ordinate and polarity of the ith event, E is the event block, and δ(·) is the Dirac function.

[0024] In one embodiment, a segmentation process is performed on each view, and all non-zero pixels are traversed to obtain a two-dimensional track area of ​​each view. The track area includes multiple track segments, including:

[0025] Use the weighted gradient operator to calculate the gradient magnitude of each view;

[0026] Sort all non-zero pixels in each view in descending order of gradient magnitude as a seed list, and use the non-zero pixel at the head of the seed list as the initial seed point;

[0027] Search the specified neighborhood of the initial seed point. If there is a non-zero pixel point and it has not been visited, add the non-zero pixel point to the end of the region list and the seed list, and set the non-zero pixel point as visited; otherwise, continue searching until the specified neighborhood of the initial seed point is traversed;

[0028] The next non-zero pixel point of the initial seed point in the seed list is taken as the next seed point, and the specified neighborhood of the next seed point is searched until the seed list is traversed to traverse all non-zero pixels on the projection surface to obtain all trajectory extraction results, and according to the area list, delete the area whose trajectory area is smaller than the preset value to obtain the two-dimensional trajectory area of ​​each view, which includes multiple trajectory segments.

[0029] In one embodiment, for multiple views, all trajectory segments of the same trajectory are merged to obtain multiple trajectory domains for each event block, including:

[0030] For two views containing the time axis in the multi-view, the track areas are arranged in descending order according to the area, and the plane area parameters of each track area are calculated;

[0031] The trajectory area with the largest area is taken as the first trajectory area, and all trajectory areas that meet the first preset condition are merged into the first trajectory area to obtain the second trajectory area;

[0032] Calculate the plane area parameters of the second trajectory area, and re-search and re-merge until it cannot be merged;

[0033] Traverse all track areas that have not been merged, perform calculations, searches, and merges to merge all track segments of the same track in two views containing the timeline;

[0034] All trajectory segments of the same trajectory in the two views containing the time axis and the trajectory regions of all views not containing the time axis are back-projected into the three-dimensional event space to obtain multiple trajectory domains for each event block.

[0035] In one embodiment, for each event block, all track domains of the same track are merged to obtain multiple track segments of each event block, including:

[0036] For each event block, calculate the XYT space parameters of all trajectory domains;

[0037] The trajectory domain with the most points is taken as the first trajectory domain, and a search is performed to merge other trajectory domains that meet the second preset condition into the first trajectory domain to obtain the second trajectory domain, and the XYT space parameters of the second trajectory domain are updated;

[0038] Re-search, merge, and update parameters until merging is impossible, traverse all unmerged track domains, and obtain multiple track segments for each event block.

[0039] In one embodiment, for all event blocks, all track segments of the same track are merged to obtain multiple tracks of the event stream, including:

[0040] Calculate the XYT space parameters of all trajectory segments in the current event block and the first n adjacent event blocks respectively;

[0041] If there are trajectory segments in the first n event blocks that meet the third preset condition with the trajectory in the current event block, then all trajectory segments of the same trajectory in the first n event blocks and the current event block are merged; if not, continue to search for trajectory segments that meet the condition that the intersection of the extended lines of the trajectory segments is located in the space-time region within the extended points of the two trajectory segments, and merge them; until all event blocks are traversed to obtain multiple trajectories of the event flow.

[0042] An event camera multi-target tracking device based on multi-view representation, comprising:

[0043] The acquisition module is used to use an event camera to shoot the motion process of multiple targets in space to obtain an event stream; the event stream is segmented to obtain event blocks;

[0044] A projection module is used to perform multi-plane projection on each event block after time realignment mapping to obtain a multi-view representation of each event block;

[0045] A segmentation module is used to perform segmentation processing on each view and traverse all non-zero pixels to obtain a two-dimensional trajectory area of ​​each view, where the trajectory area includes multiple trajectory segments;

[0046] A merging module is used to merge all the track segments of the same track for multiple views to obtain multiple track domains for each event block; for each event block, merge all the track domains of the same track to obtain multiple track segments for each event block; for all event blocks, merge all the track segments of the same track to obtain multiple tracks of the event stream;

[0047] The tracking module is used to complete multi-target tracking based on multiple trajectories of the event stream.

[0048] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0049] The event camera is used to shoot the motion process of multiple targets in space to obtain an event stream; the event stream is segmented to obtain event blocks;

[0050] For each event block, after time realignment mapping, multi-plane projection is performed to obtain a multi-view representation of each event block;

[0051] For each view, segmentation processing is performed, and all non-zero pixels are traversed to obtain a two-dimensional trajectory area for each view, and the trajectory area includes multiple trajectory segments;

[0052] For multiple views, all the track segments of the same track are merged to obtain multiple track domains for each event block; for each event block, all the track domains of the same track are merged to obtain multiple track segments for each event block; for all event blocks, all the track segments of the same track are merged to obtain multiple tracks of the event stream;

[0053] Based on multiple trajectories of the event stream, multi-target tracking is completed.

[0054] The above-mentioned event camera multi-target tracking method, device and equipment based on multi-view representation use the time and space information contained in the event for three-plane representation, and combine the high temporal resolution characteristics of the event camera to transform the target tracking problem into a trajectory segmentation problem. An event camera target tracking structure that can apply traditional image frame algorithms is constructed, and accurate target tracking results are obtained by using the complementary trajectory information of different planes. The measurement method is accurate and efficient, the process is simple and convenient, and can be widely used in related target tracking, target segmentation and other tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is an application scenario diagram of an event camera multi-target tracking method based on multi-view representation in one embodiment;

[0056] Figure 2 is a flow chart of a method for tracking multiple targets using an event camera based on multi-view representation in one embodiment;

[0057] Figure 3 is a schematic diagram based on three-plane representation in one embodiment;

[0058] Figure 4 is a structural block diagram of an event camera multi-target tracking device based on multi-view representation in one embodiment;

[0059] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0061] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0062] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "multiple groups" means at least two groups, such as two groups, three groups, etc., unless otherwise clearly and specifically defined.

[0063] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0064] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0065] The event camera multi-target tracking method based on multi-view representation provided in this application can be applied to Figure 1 In the application scenario diagram shown, the terminal 102 communicates with the server 104 through the network, and the terminal 102 may include but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 may be various portal websites, servers corresponding to the backend of the work system, etc.

[0066] This application provides an event camera multi-target tracking method based on multi-view representation, such as Figure 2 The flowchart shown in FIG. 1 shows a method for applying the method to Figure 1 The following is an example of a terminal in the example, including:

[0067] Step 202: Use an event camera to shoot the motion process of multiple targets in space to obtain an event stream; segment the event stream to obtain event blocks.

[0068] In this step, segmentation includes segmentation according to a fixed time interval or a fixed number of events.

[0069] Step 204 , for each event block, after time realignment mapping, multi-plane projection is performed to obtain a multi-view representation of each event block.

[0070] Specifically:

[0071] Select three orthogonal planes XOY, YOT, TOX in a three-dimensional Cartesian rectangular coordinate system as multi-view projection planes, the multi-view projection planes including: an XOY view projection plane, a YOT view projection plane, and a TOX view projection plane;

[0072] Perform time realignment mapping on all events in each event block to obtain a realigned event block, so as to obtain a denser event distribution in terms of time sequence.

[0073] The realigned event blocks are projected onto the multi-view projection plane (the realigned event blocks are projected onto the multi-view projection plane in the same way as ordinary event frames without considering polarity), and passed through a multi-scale Gaussian pyramid (multi-scale fusion processing is performed using a pyramid with Gaussian sampling) to obtain a multi-view representation of each event block, which is a projection view on three planes.

[0074] Among them, for each event block, time realignment mapping is performed according to the time corresponding function:

[0075]

[0076] Where T′ is the pseudo timestamp set, T is the original timestamp set of the event block, and t i ' is a component of T', i.e., a pseudo-timestamp, t i The constituent element of T is the original timestamp, t i ′ and t i One-to-one correspondence, g(·) is a re-labeled function, which belongs to the time correspondence function;

[0077] Take the projection of the realignment event block onto the YOT view projection plane as an example:

[0078]

[0079] In the formula, e i is the ith event, (t i ,x i ,y i ,p i ) are the timestamp, abscissa, ordinate and polarity of the ith event, E is the event block, and δ(·) is the Dirac delta function.

[0080] In this step, the three planes are represented as Figure 3 As shown in the figure, after the time realignment mapping, the event block is directly projected to the XOY, YOT and TOX planes, and then multi-scale fusion is performed through the Gaussian pyramid to output a multi-view representation.

[0081] The event blocks are mapped through time realignment, which can solve the problem in the prior art that clustering parameters are difficult to confirm when the spans of targets with different moving speeds in the time dimension are inconsistent.

[0082] Step 206 , segment each view and traverse all non-zero pixels to obtain a two-dimensional track area of ​​each view, where the track area includes multiple track segments.

[0083] Specifically:

[0084] Use the Gaussian weighted gradient operator to calculate the gradient magnitude of each view;

[0085] Sort all non-zero pixels in each view in descending order of gradient magnitude as a seed list, and use the non-zero pixel at the head of the seed list (i.e., the pixel with the largest gradient magnitude) as the initial seed point;

[0086] Search the specified neighborhood of the initial seed point. If there is a non-zero pixel point and it has not been visited (unvisited means not searched), add the non-zero pixel point to the end of the region list (region list refers to the set of regions where the trajectory is located) and the seed list (seed list refers to the set of seed points), and set the non-zero pixel point as visited; otherwise, continue searching until the specified neighborhood of the initial seed point is traversed;

[0087] The next non-zero pixel point of the initial seed point in the seed list is taken as the next seed point, and the specified neighborhood of the next seed point is searched until the seed list is traversed to traverse all non-zero pixels of the projection surface to obtain all trajectory extraction results, and according to the area list, the area whose trajectory area is smaller than a preset value (the preset value can be determined according to the prior art) is deleted to obtain a two-dimensional trajectory area for each view, and the trajectory area includes multiple trajectory segments.

[0088] In this step, the segmentation process is only an embodiment, and other segmentation process methods in the prior art may also be used.

[0089] Step 208: for multiple views, merge all the track segments of the same track to obtain multiple track domains for each event block; for each event block, merge all the track domains of the same track to obtain multiple track segments for each event block; for all event blocks, merge all the track segments of the same track to obtain multiple tracks of the event stream.

[0090] Specifically:

[0091] For multiple views, all track segments of the same track are merged to obtain multiple track domains of each event block, including: for two views (YOT and TOX) containing the time axis in the multiple views, the track areas are arranged in descending order according to the area, and the plane area parameters of each track area are calculated; the track area with the largest area is taken as the first track area, and according to the plane area parameters, all track areas that meet the first preset condition are merged into the first track area to obtain the second track area; the plane area parameters of the second track area are calculated, and the track areas that are not merged are re-searched and re-merged until they cannot be merged; all track areas that have not been merged are traversed, and calculation, search, and merging are performed to merge all track segments of the same track in the two views containing the time axis; all track segments of the same track in the two views containing the time axis and the track areas of all views not containing the time axis are back-projected into the three-dimensional event space to obtain a track set composed of event points, each track in the three plane track sets is traversed, and the intersection of two tracks is taken and saved as a new track segment to fuse the results of different views to obtain multiple track domains of each event block. Taking the example that the plane area parameters include the centroid, direction vector and area width of the trajectory area, the trajectory area waiting for other areas to be merged is taken as the merged area, and the trajectory areas other than the merged area are taken as the to-be-merged area. The first preset condition is that the distance between the centroid of the to-be-merged area and the straight line where the direction vector of the merged area is located is less than or equal to the area width of the merged area. Specifically: the centroid, direction vector and area width of each trajectory area are calculated; the trajectory area with the largest area is taken as the first trajectory area, and a straight line is formed with the centroid of the first trajectory area as the starting point and the direction vector as the direction, and a search is performed to merge all trajectory areas whose distance between the centroid and the straight line is not greater than the area width of the first trajectory area into the first trajectory area to obtain the second trajectory area; the centroid, direction vector and area width of the second trajectory area are calculated, and the search and re-merge are performed until the merging is impossible.

[0092] For each event block, all trajectory domains of the same trajectory are merged to obtain multiple trajectory segments of each event block, including: for each event block, the XYT space parameters of all trajectory domains are calculated; the trajectory domain with the largest number of points is taken as the first trajectory domain, and the search is performed according to the XYT space parameters, and other trajectory domains that meet the second preset condition are merged into the first trajectory domain to obtain the second trajectory domain, and the XYT space parameters of the second trajectory domain are calculated; the search, merging, and updating parameters are performed again until merging is impossible, and all trajectory domains that have not been merged are traversed to obtain multiple trajectory segments of each event block. Taking the XYT space parameters including the centroid, direction vector and area width of the trajectory domain as an example, the second preset condition is that the distance between the two trajectory domains is less than the area width of any trajectory domain. Specifically: the centroid c, direction vector d and area width w of all trajectory domains in the current event block are calculated, the trajectory domain with the largest number of points is used as the initial area (i.e., the first trajectory domain), the centroid is used as the starting point, and the direction vector is used as the direction to form a straight line, and all trajectory domains whose distance from the current trajectory domain is not greater than the width of the trajectory domain are merged until all trajectory domains have been merged; the trajectory domains whose direction difference is less than a threshold (the threshold can be preset according to the prior art) are merged to obtain the complete event trajectory of the current event block.

[0093] For all event blocks, all trajectory segments of the same trajectory are merged to obtain multiple trajectories of the event stream, including: respectively calculating the XYT partial spatial parameters of all trajectory segments in the current event block and the first n adjacent event blocks; if there is a trajectory segment in the first n event blocks that satisfies the third preset condition with the trajectory segment in the current event block (the third preset condition is: the change angle of the trajectory segments in the current event block and the first n adjacent event blocks is less than a preset range), then merge all trajectory segments of the same trajectory in the first n event blocks and the current event block; if not satisfied, continue to search for trajectory segments that meet the condition that the intersection of the trajectory segment extension line is located in the space-time region within the extension points of the two trajectory segments, and merge them; until all event blocks are traversed to obtain multiple trajectories of the event stream. Specifically: calculate the centroid and direction vector of all trajectories Current appearing in the current event block; calculate the centroid and direction vector of all trajectories Previous appearing in the adjacent first n event blocks; associate the trajectories in Current with the trajectories in Previous, and form a straight line with the centroid as the starting point and the direction vector as the direction. When there is a trajectory with the smallest direction change and not greater than a certain value (the constant value can be preset according to the prior art), merge it. If there is still no corresponding trajectory in Current, then combine the trajectory with the remaining trajectories in Previous to determine whether there is an intersection of the trajectory extension line located in the time and space region of the two trajectory extension points. If so, merge it. If not, save the current trajectory as a new trajectory. It should be noted that the idea of ​​divide and conquer is used in searching the first n event blocks here. If the same trajectory can be found in the first n event blocks, it means that the current trajectory belongs to a previously appeared trajectory; if not, it means that the trajectory is newly appeared; therefore, searching and merging in the current and previous event blocks, each time processing the current event block, can associate the current and previous trajectories.

[0094] In this step, multiple views representing different information are fused, which solves the problem in the prior art that it is easy to detect and track errors when different trajectories partially overlap or intersect.

[0095] Step 210, completing multi-target tracking according to multiple trajectories of the event stream.

[0096] In this step, multiple trajectories of the event stream correspond to the motion processes of multiple targets in space, thereby completing multi-target tracking.

[0097] The above-mentioned event camera multi-target tracking method based on multi-view representation uses the time and space information contained in the event for three-plane representation. Combined with the high temporal resolution characteristics of the event camera, the target tracking problem is transformed into a trajectory segmentation problem. It is adapted to a variety of image segmentation algorithms and constructs an event camera target tracking structure that can apply traditional image frame algorithms. The trajectory information of different planes is complementary to obtain accurate target tracking results. The measurement method is accurate and efficient, the process is simple and convenient, and it can be widely used in related target tracking, target segmentation and other tasks.

[0098] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0099] The present application also provides an event camera multi-target tracking device based on multi-view representation, such as Figure 4 As shown, in one embodiment, it includes: an acquisition module 402, a projection module 404, a segmentation module 406, a merging module 408 and a tracking module 410, wherein:

[0100] The acquisition module 402 is used to use an event camera to shoot the motion process of multiple targets in space to obtain an event stream; segment the event stream to obtain event blocks;

[0101] A projection module 404 is used to perform multi-plane projection on each event block after time realignment mapping to obtain a multi-view representation of each event block;

[0102] The segmentation module 406 is used to perform segmentation processing on each view and traverse all non-zero pixels to obtain a two-dimensional track area of ​​each view, where the track area includes multiple track segments;

[0103] A merging module 408 is used to merge all track segments of the same track for multiple views to obtain multiple track domains for each event block; merge all track domains of the same track for each event block to obtain multiple track segments for each event block; merge all track segments of the same track for all event blocks to obtain multiple tracks of the event stream;

[0104] The tracking module 410 is used to complete multi-target tracking according to multiple trajectories of the event stream.

[0105] For the specific definition of the event camera multi-target tracking device based on multi-view representation, please refer to the definition of the event camera multi-target tracking method based on multi-view representation above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0106] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an event camera multi-target tracking method based on multi-view representation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0107] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0109] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An event camera multi-target tracking method based on multi-view representation, characterized in that: include: Use event cameras to capture the motion process of multiple targets in space and obtain event streams; Split the event stream to obtain event blocks; For each event block, after time realignment mapping, multi-plane projection is performed to obtain a multi-view representation of each event block; For each view, segmentation processing is performed, and all non-zero pixels are traversed to obtain a two-dimensional trajectory area for each view, and the trajectory area includes multiple trajectory segments; For multiple views, all trajectory segments of the same trajectory are merged to obtain multiple trajectory domains for each event block; For each event block, all trajectory domains of the same trajectory are merged to obtain multiple trajectory segments for each event block; For all event blocks, merge all trajectory segments of the same trajectory to obtain multiple trajectories of the event stream; Based on multiple trajectories of the event stream, multi-target tracking is completed.

2. The event camera multi-target tracking method based on multi-view representation according to claim 1, characterized in that: For each event block, after time realignment mapping, multi-plane projection is performed to obtain a multi-view representation of each event block, including: Select three orthogonal planes in a three-dimensional Cartesian rectangular coordinate system as multi-view projection planes; For each event block, time realignment mapping is performed to obtain a realigned event block; The realigned event blocks are projected onto the multi-view projection plane and passed through a multi-scale Gaussian pyramid to obtain a multi-view representation of each event block.

3. The event camera multi-target tracking method based on multi-view representation according to claim 2, characterized in that: For each event block, time realignment mapping is performed, including: Where T′ is the pseudo timestamp set, T is the original timestamp set of the event block, and t i ' is the constituent element of T', t i are the components of T, and g(·) is the relabeling function.

4. The event camera multi-target tracking method based on multi-view representation according to claim 3, characterized in that: Projecting the realignment event block onto the multi-view projection plane includes: projecting the realignment event block onto the XOY view projection plane, the YOT view projection plane, and the TOX view projection plane; Among them, projecting the realignment event block onto the YOT view projection plane includes: In the formula, e i is the ith event, (t i ,x i ,y i ,p i ) are the timestamp, abscissa, ordinate and polarity of the ith event, E is the event block, and δ(·) is the Dirac function.

5. The event camera multi-target tracking method based on multi-view representation according to any one of claims 1 to 4, characterized in that: For each view, segmentation is performed and all non-zero pixels are traversed to obtain a two-dimensional track area for each view. The track area includes multiple track segments, including: Use the weighted gradient operator to calculate the gradient magnitude of each view; Sort all non-zero pixels in each view in descending order of gradient magnitude as a seed list, and use the non-zero pixel at the head of the seed list as the initial seed point; Search the specified neighborhood of the initial seed point. If there is a non-zero pixel point and it has not been visited, add the non-zero pixel point to the end of the region list and the seed list, and set the non-zero pixel point as visited; otherwise, continue searching until the specified neighborhood of the initial seed point is traversed; The next non-zero pixel point of the initial seed point in the seed list is taken as the next seed point, and the specified neighborhood of the next seed point is searched until the seed list is traversed to traverse all non-zero pixels on the projection surface to obtain all trajectory extraction results, and according to the area list, delete the area whose trajectory area is smaller than the preset value to obtain the two-dimensional trajectory area of ​​each view, which includes multiple trajectory segments.

6. The event camera multi-target tracking method based on multi-view representation according to any one of claims 1 to 4, characterized in that: For multiple views, all trajectory segments of the same trajectory are merged to obtain multiple trajectory domains for each event block, including: For two views containing the time axis in the multi-view, the track areas are arranged in descending order according to the area, and the plane area parameters of each track area are calculated; The trajectory area with the largest area is taken as the first trajectory area, and all trajectory areas that meet the first preset condition are merged into the first trajectory area to obtain the second trajectory area; Calculate the plane area parameters of the second trajectory area, and re-search and re-merge until it cannot be merged; Traverse all track areas that have not been merged, perform calculations, searches, and merges to merge all track segments of the same track in two views containing the timeline; All trajectory segments of the same trajectory in the two views containing the time axis and the trajectory regions of all views not containing the time axis are back-projected into the three-dimensional event space to obtain multiple trajectory domains for each event block.

7. The event camera multi-target tracking method based on multi-view representation according to any one of claims 1 to 4, characterized in that: For each event block, all trajectory domains of the same trajectory are merged to obtain multiple trajectory segments for each event block, including: For each event block, calculate the XYT space parameters of all trajectory domains; The trajectory domain with the most points is taken as the first trajectory domain, and a search is performed to merge other trajectory domains that meet the second preset condition into the first trajectory domain to obtain the second trajectory domain, and the XYT space parameters of the second trajectory domain are updated; Re-search, merge, and update parameters until merging is impossible, traverse all unmerged track domains, and obtain multiple track segments for each event block.

8. The event camera multi-target tracking method based on multi-view representation according to any one of claims 1 to 4, characterized in that: For all event blocks, all trajectory segments of the same trajectory are merged to obtain multiple trajectories of the event stream, including: Calculate the XYT space parameters of all trajectory segments in the current event block and the first n adjacent event blocks respectively; If there are trajectory segments in the first n event blocks that meet the third preset condition with the trajectory in the current event block, then all trajectory segments of the same trajectory in the first n event blocks and the current event block are merged; if not, continue to search for trajectory segments that meet the condition that the intersection of the extended lines of the trajectory segments is located in the space-time region within the extended points of the two trajectory segments, and merge them; until all event blocks are traversed to obtain multiple trajectories of the event flow.

9. An event camera multi-target tracking device based on multi-view representation, characterized in that: include: An acquisition module is used to capture the motion process of multiple targets in space using an event camera to obtain an event stream; Split the event stream to obtain event blocks; A projection module is used to perform multi-plane projection on each event block after time realignment mapping to obtain a multi-view representation of each event block; A segmentation module is used to perform segmentation processing on each view and traverse all non-zero pixels to obtain a two-dimensional trajectory area of ​​each view, where the trajectory area includes multiple trajectory segments; The merging module is used to merge all trajectory segments of the same trajectory in multiple views to obtain multiple trajectory domains for each event block; For each event block, all trajectory domains of the same trajectory are merged to obtain multiple trajectory segments for each event block; For all event blocks, merge all trajectory segments of the same trajectory to obtain multiple trajectories of the event stream; The tracking module is used to complete multi-target tracking based on multiple trajectories of the event stream.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images