A multi-camera target detection and tracking method and apparatus

Abstract

The application discloses a multi-camera target detection and tracking method, which comprises the following steps: acquiring at least two video stream image data from different cameras, wherein each video stream image data comprises at least image data of an overlapping area; acquiring visual map information of a position of the overlapping area in the video stream image data; converting feature point information extracted from each image frame of each video stream image data to the visual map, so as to fuse the feature point information of each image frame to the same visual angle and obtain fusion information under the same visual angle, wherein each image frame has contemporaneity; and performing target detection and tracking based on the fusion information. The application does not need to generate a single-camera track and then perform track fusion, solves the problem that it is difficult to distinguish and select multiple single-camera tracks when performing fusion, avoids the calculation power consumption caused by track fusion, and improves the accuracy and reliability of target detection and tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image-based target detection, in particular, to a multi-camera target detection and tracking method. BACKGROUND

[0002] With the continuous progress of society, the application range of target detection and tracking is becoming wider and wider. For example, smart city or smart intersection, autonomous driving and other application scenarios usually need to fuse the information of multiple images from multiple cameras and use the fused images for target detection and tracking.

[0003] At present, most target detection and tracking methods are based on target detection results, and the target position is accurately located in each frame of the continuous frames of the video stream, which can be understood as target detection and tracking. Multi-camera target detection and tracking respectively detects and tracks the image information from each camera, generates a single-camera trajectory, and then projects the trajectory in each camera into the same perspective view, such as the Bird Eye View (BEV) view, and finally fuses the projected trajectories into a global trajectory.

[0004] The above-mentioned multi-camera target detection and tracking method is a step-by-step target detection and tracking method, which makes the system more complex and occupies communication resources. Moreover, if there is a problem with the trajectory in a single camera, it is difficult to distinguish and select the trajectory in a single camera when the trajectories of multiple cameras are fused in the same perspective, resulting in difficulty in obtaining the correct global trajectory. SUMMARY

[0005] The present application provides a multi-camera target detection and tracking method to obtain an accurate global trajectory.

[0006] The present application provides a multi-camera target detection and tracking method, which comprises:

[0007] Obtaining at least two or more video stream image data from different cameras, wherein each video stream image data comprises at least image data of an overlapping area,

[0008] Obtaining visual map information of the position corresponding to the space of the video stream image data,

[0009] Converting the feature point information extracted from each image frame of each video stream image data into a visual map to fuse the feature point information of each image frame into the same perspective to obtain fusion information in the same perspective, wherein each image frame in each image frame has contemporaneity,

[0010] Based on the fusion information, target detection and tracking are performed.

[0011] Preferably, the converting the feature point information extracted from each image frame of each video stream into the visual map comprises:

[0012] For each image frame in each image frame:

[0013] performing feature extraction on the image frame respectively to obtain feature point information and / or a feature map of the image frame,

[0014] projecting the feature points of the image frame into the visual map using a projection matrix of a camera from which the image frame is derived to obtain projection feature point information of the image frame under the same perspective,

[0015] determining the projection feature point information of all image frames in the image frame as the fusion information,

[0016] wherein,

[0017] the projection matrix is used to represent a projection matrix of a mapping relationship between a pixel point in a camera image and a spatial point in a visual map;

[0018] the target detection and tracking based on the fusion information comprises:

[0019] performing target detection based on the projection feature point information of all image frames in the image frame to obtain target detection results under the same perspective corresponding to all image frames in the image frame, and determining the target detection results as target detection results of an image frame group of the image frame,

[0020] obtaining target trajectory data from the target detection results of each image frame group,

[0021] wherein, the target detection results of each image frame group are target detection results under the same perspective corresponding to all image frames in the image frame at different times.

[0022] Preferably, the obtaining target trajectory data from the target detection results of each image frame group comprises:

[0023] labeling each target detection result in the intersection of the target detection results of the current image frame group and the target detection results of the historical image frame group,

[0024] obtaining target trajectory data in a world coordinate system from the labeled target detection results of each image frame group.

[0025] Preferably, the performing target detection based on the projection feature point information of all image frames in the image frame comprises:

[0026] search the pre-initialized target feature based on the projection feature point information to obtain target reference position information,

[0027] back-project the target reference position information into each image frame by using the projection matrix of the camera from which each image frame is derived to determine position information of the target reference position in the image frame,

[0028] obtain the feature of the target according to the position information of the image frame to obtain the feature of the target in each channel,

[0029] fuse the feature of the target in each channel to obtain the fused feature of the target,

[0030] search the fused feature based on the projection feature point information to obtain the target detection result,

[0031] The target detection result includes global position information, target size, and confidence in the same view angle.

[0032] Preferably, the searching of the pre-initialized target feature based on the projection feature point information to obtain the target reference position information comprises:

[0033] inputting the projection feature point information and the pre-initialized target detection vector into a machine learning model to obtain the reference position information of each target, wherein the target detection vector includes target feature vectors of two or more targets,

[0034] The back-projecting of the target reference position information into each image frame by using the projection matrix of the camera from which each image frame is derived comprises:

[0035] back-projecting the reference position information of each target into the feature map corresponding to each image frame by using the projection matrix of the camera from which each video stream image is derived to determine the position information of the reference position of each target in the feature map,

[0036] The obtaining of the feature of the target according to the position information of the image frame to obtain the feature of the target in each channel comprises:

[0037] obtaining the feature corresponding to each target according to the position information in the feature map of each target.

[0038] Preferably, the fusing of the feature of the target in each channel to obtain the fused feature of the target comprises:

[0039] performing feature fusion of each target to obtain the fused feature of each target,

[0040] The searching of the fused feature based on the projection feature point information comprises:

[0041] Input the fusion feature of each target and the projection feature point information into the machine learning model to obtain the target detection result.

[0042] Preferably, the feature fusion of each target is performed respectively, including:

[0043] For each target:

[0044] Based on each feature map, fuse the features of the target in each feature map to obtain a first fusion feature,

[0045] Based on each feature map, fuse the features of other targets except the same target in each feature map to obtain a second fusion feature,

[0046] Fuse the first fusion feature and the second fusion feature to obtain the fusion feature of the target;

[0047] The method further includes:

[0048] According to the set confidence threshold, filter the target detection results of the current image frame group to obtain effective target detection results,

[0049] Add the effective target detection results to the initialization target feature set of the next image frame group.

[0050] Preferably, the target detection results in the intersection of the target detection results of the current image frame group and the target detection results of the historical image frame group are marked, including:

[0051] If the effective target detection result of the current image frame group comes from the initialization target feature, a new tracking identifier is assigned to the effective target detection result;

[0052] If the effective target detection result of the current image frame group comes from the effective target detection result added by the previous image frame group, the tracking identifier of the effective target detection result added by the previous image frame group is used;

[0053] The target trajectory data in the world coordinate system is obtained from the target detection results marked by each image frame group, including:

[0054] The target detection results with the same tracking identifier from the target detection results marked by each image frame group are determined as the target trajectory data of the target detection result;

[0055] The visual map is an aerial view perspective map, and the same perspective is an aerial view perspective.

[0056] The embodiments of the present application also provide a multi-camera target detection and tracking device, which includes:

[0057] The first acquisition module is configured to acquire video stream image data from more than two cameras,

[0058] The second acquisition module is configured to acquire visual map information of a position corresponding to the video stream image data,

[0059] The target detection and tracking module is configured to convert feature point information extracted from each image frame of each video stream image data to a visual map, so as to fuse the feature point information of each image frame to the same perspective, and obtain fusion information under the same perspective, wherein each image frame has contemporaneity.

[0060] Target detection and tracking are performed based on the fusion information.

[0061] The embodiments of the present application further provide a computer readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any of the multi-camera target detection and tracking methods.

[0062] The multi-camera target detection and tracking method provided by the embodiments of the present application projects video stream image features from different cameras to the same perspective, and performs target detection and tracking based on fusion information under the same perspective. In this way, information fusion is performed from the source, which is beneficial to improve the accuracy of information sources for target detection and tracking, improve the intelligence of multi-camera fusion, and solve the problem of difficult discrimination and selection when multiple single-camera trajectories are fused. In addition, the algorithm consumption caused by trajectory fusion is avoided, and the accuracy and reliability of target detection and tracking are improved. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 FIG. 1 is a flowchart of a multi-camera target detection and tracking method according to an embodiment of the present application.

[0064] Figure 2 FIG. 2 is a flowchart of a multi-camera target detection and tracking method in a specific scene according to an embodiment of the present application.

[0065] Figure 3 FIG. 3 is a schematic diagram of four cameras collecting a frame of video stream images of four directions of an intersection.

[0066] Figure 4 FIG. 4 is a schematic diagram of a high-precision map of a bird's-eye view of a traffic intersection.

[0067] Figure 5 FIG. 5 is a schematic diagram of a target detection process.

[0068] Figure 6A schematic diagram of a multi-camera target detection and tracking device according to an embodiment of the present application.

[0069] Figure 7 Another schematic diagram of a multi-camera target detection and tracking device according to an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the purposes, technical means and advantages of the present application more clear, the present application is further described in detail below with reference to the accompanying drawings.

[0071] According to an embodiment of the present application, feature points of image frames corresponding to the same time in each video stream image data are projected to a same perspective visual map, so as to fuse the feature point information of the image frames in each video stream image to the same perspective, and perform target detection and tracking based on the fused information under the same perspective.

[0072] Referring to Figure 1 as shown, Figure 1 A flowchart of a multi-camera target detection and tracking method according to an embodiment of the present application. The method comprises:

[0073] Step 101, acquiring at least two or more video stream image data,

[0074] Each video stream image data comprises at least image data of the same scene collected from different shooting angles. Usually, each video stream image data is obtained by a camera installed at different positions collecting image data of the same scene, so that image data from different shooting angles can be obtained.

[0075] From the perspective of data, the same scene means that there is at least intersection data between each video stream image data. From the perspective of spatial position, the same scene means a target set located in the same spatial position range. The spatial position range can be set according to needs. In other words, there is an overlapping area between each video stream image.

[0076] The acquisition approach can be real-time video stream image data acquired from a camera, or non-real-time video stream image data acquired from a storage end. The present application does not limit this.

[0077] Step 102, acquiring visual map information of a position where the overlapping area is located in the video stream image data,

[0078] As an example, the position where the overlapping area is located can be acquired through position information of the camera. According to geographical position information of the camera installation, corresponding bird's-eye perspective visual map information is acquired from a map library.

[0079] The visual map can be a bird's eye view visual map, the bird's eye view can be understood as a top view, the bird's eye view visual map information is equivalent to top view visual map information, and the map information includes global position information in a world coordinate system.

[0080] In step 103, the feature point information extracted from each image frame of each video stream image data is converted into the same perspective visual map, so as to fuse the feature point information of each image frame into the same perspective, and obtain the fusion information in the same perspective.

[0081] Each image frame in each image frame has contemporaneity, for example, the image frames of each path correspond to the same time, and it should be understood that the image frames of the same time in the absolute sense are beneficial to improve the accuracy of target detection and tracking, for example, each image frame has the same timestamp information, but in actual application, each image frame in each image frame does not need to be strictly at the same moment, as long as the time difference between each image frame in each image frame is within a set time threshold, which is equivalent to corresponding to the same time and has contemporaneity. If each image frame does not have contemporaneity, synchronization processing can be performed.

[0082] In this step, for each single-frame image frame in each image frame:

[0083] Feature extraction is performed on the image frame respectively, to obtain feature point information and / or feature map of the image frame.

[0084] The projection matrix of the camera from which the image frame is derived is used to project the feature points of the image frame into the visual map, to obtain the projection feature point information of the image frame in the same perspective, and the projection feature point information includes global position information.

[0085] The projection feature point information of all image frames in each image frame is determined as the fusion information, so that the fusion information represents the feature information of the scene at the same time in the same perspective, that is, the feature information corresponding to all image frames with contemporaneity in the same perspective. In view of the fact that each image frame is a set of image frames with a time difference within a set time threshold, that is, the image frames with contemporaneity, the set is referred to as an image frame group in the present application, and the fusion information can be understood as the fusion information of the image frame group.

[0086] In the step 103, the feature point information extracted from each image frame of each video stream image data is converted into the same perspective visual map, so as to fuse the feature point information of each image frame into the same perspective, and obtain the fusion information in the same perspective.

[0087] The projection matrix is used to represent the mapping relationship between the pixel points in the camera image and the space points in the visual map, and different cameras correspond to different projection matrices.

[0088] In step 104, target detection and tracking are performed based on the fusion information.

[0089] In this step, target detection is performed based on the projection feature point information of all image frames in each image frame. As an example, target features initialized in advance are searched based on the projection feature point information, target reference position information is obtained, the target reference position information is back-projected into each image frame using the projection matrix of the camera from which the image frame is derived to determine the position information of the target reference position in the image frame, the features of the target in each image frame are obtained according to the position information of the image frame, the features of the target in each image frame are fused to obtain fused features of the target, the fused features are searched based on the projection feature point information, and target detection results corresponding to the same perspective of all image frames in each image frame are obtained. The current detection result is determined as the target detection result of the image frame group of each image frame, wherein the image frame group is composed of each image frame with simultaneity; and the target detection result includes global position information in the same perspective, target size, and confidence. The above detection method of searching the fused features can realize the correlation detection of multi-image frame feature information, and is beneficial to improving the accuracy of target detection.

[0090] Target trajectory data is obtained from the target detection results of each image frame group, wherein the target detection results of each image frame group are the target detection results corresponding to the image frame group at different times, i.e., the target detection results of the historical image frame group. As an example, if the fused information is the fused information of one image frame group, target detection and tracking can be performed based on the fused information of the current image frame group.

[0091] Taking target detection and tracking based on the fused information of multiple image frame groups as an example, each target detection result in the intersection of the target detection results in the same perspective of the current image frame group and the target detection results in the same perspective of the historical image frame group is marked, for example, the target detection results in the intersection inherit the existing tracking identifier, and the target detection results in the same perspective of the current image frame group that are not in the intersection are assigned new tracking identifiers; and the target detection results with the same tracking identifier are determined as the target trajectory data of the target detection result. Thus, target detection and tracking are integrated, and it is not necessary to perform detection first and then tracking, so that the overall detection and tracking is more concise.

[0092] The multi-camera target detection and tracking method of the embodiments of the present application integrates each image frame feature point information into the same perspective, provides an end-to-end, multi-perspective, detection and tracking integrated method, avoids the problem that if there is a problem in the single camera trajectory, it is difficult to obtain the correct global trajectory, and is beneficial to improving the reliability and accuracy of multi-camera target detection and tracking.

[0093] For the convenience of understanding the embodiments of the present application, the following will be described by taking the multi-camera target detection and tracking applied to the traffic intersection as an example, and it should be understood that the present application is not limited to the multi-camera target detection and tracking applied to the traffic intersection, and any multi-camera target detection and tracking can be applied, such as the multi-camera installed on the vehicle body for detecting and tracking the surrounding targets.

[0094] Referring to Figure 2 as shown, Figure 2 is a flowchart of the multi-camera target detection and tracking method in the specific scenario of the embodiments of the present application. The method comprises the following steps:

[0095] In step 201, video stream images from the multi-camera and visual map information corresponding to the spatial position of the video stream images are acquired,

[0096] As an example, the video stream images from the multi-camera can be, for example, four cameras installed at a traffic intersection to collect video stream images of four directions of the intersection with overlapping areas, and four video stream images are obtained. Referring to Figure 3 as shown, Figure 3 is a schematic diagram of one frame of video stream images collected by the four cameras in four directions of the intersection.

[0097] As an example, the visual map information corresponding to the spatial position of the video stream images can be the map information of the traffic intersection, so as to obtain the global position information of the video stream images. The position information in the map information can be described by using the global coordinate information in the world coordinate system.

[0098] For the convenience of presentation and display, the map information can be a bird's eye view high-precision map. Referring to Figure 4 as shown, Figure 4 is a schematic diagram of a bird's eye view high-precision map at the traffic intersection. It should be understood that the map information can also be a general map.

[0099] The high-precision map is a kind of thematic map relative to the ordinary navigation electronic map, also known as high-resolution map. Its absolute position accuracy is close to several meters, and the relative position accuracy is in the centimeter level. The data organization mode is to describe the information of water system, railway, block, building, traffic marking line, etc. through different layers, and then the layers are superimposed to express.

[0100] In step 202, the projection matrix corresponding to the visual map information of the camera from which each video stream image is derived is acquired,

[0101] In view of the mapping relationship between the pixel points and the spatial points in a frame of camera image satisfying a camera model, the mapping relationship can be described by a projection matrix, so that a plurality of pixel points with landmarks are selected in the camera image, or a plurality of spatial points with landmarks are selected in the map information, and the projection matrix can be calculated by using the pixel coordinates and the corresponding map information, and the internal and external calibration parameters of the camera can be obtained in advance.

[0102] Referring to Figure 5 as shown, Figure 5 is a schematic diagram of the selected pixel points in a frame of image and the corresponding spatial points, wherein the left image is a camera image, and the right image is a map point image. As an example, the pixel points corresponding to the traffic marker points are selected in the camera image, the map information corresponding to the traffic marker points can be determined in the high-precision map, and the projection matrix of the camera from which the image is derived and the high-precision map is calculated by using the pixel coordinates and the map coordinates, wherein the projection matrix is calculated in the following manner: the coordinates of a plurality of corresponding points of the camera image and the high-precision map are substituted into the constructed linear equation set, and the least square method is used to solve and obtain, for example, the direct linear transformation (DLT) algorithm, the P3P algorithm, the EPnP algorithm, and the bundle adjustment (BA) algorithm.

[0103] The projection matrix can be determined and stored in an offline manner or in real time.

[0104] In step 203, for each video stream image, the features in the current frame are extracted respectively to obtain the current feature map and / or the current feature information of each channel,

[0105] In this step, the CNN neural network can be used to extract the features in the current frame of each channel, and the feature data can be organized according to the pixel position information of the features to obtain the corresponding current feature map, wherein the current frame is a single frame image of each channel of the video stream at the same time.

[0106] In step 204, for each current feature map, the following processing is performed:

[0107] In step 2041, the projection matrix of the camera from which the video stream image is derived is used to project the feature points in the current feature map of the channel to the BEV visual map to obtain the projection feature point position information, and the target feature set is initialized in the BEV visual map;

[0108] In this step, the inner product of the pixel coordinate vector of the feature point and the camera projection matrix can be used to obtain the global position information of the projection feature point in the world coordinate system;

[0109] As an example, a set of target features for target search is initialized, each target feature being a vector of a set length, for example, each target feature is set for a motor vehicle and a non-motor vehicle target respectively, and all target features are taken as a set of target detection vectors for initialization to obtain an initialized target feature vector.

[0110] The target detection vector can be a 3D target detection vector including 3D information, or a 2D target detection vector including 2D information.

[0111] In step 2042, based on the initialized target feature, reference position information of each target in the BEV visual map is obtained.

[0112] In this step, based on the projected feature points of the BEV visual map, target search is performed to obtain the reference position information of each target detected in the BEV visual map,

[0113] The initialized target detection vector and the projected feature point position information are input into a machine learning model, for example, a multi-layer perceptron, and the multi-layer perceptron is used to search for target detection vectors for the projected feature points of the BEV visual map, and the reference position information of each target in the BEV visual map is analyzed. For example, the reference position information of different vehicles in the BEV visual map is analyzed.

[0114] In step 2043, the reference position information of each target is back-projected into the current feature map of the road using the projection matrix of the camera to determine the feature position of the reference position of each target in the current feature map of the road, and the corresponding feature is obtained based on the feature position.

[0115] In this step, for each target reference position, the feature position information in the current feature map corresponding to the reference position information is obtained according to the projection matrix of the camera and the reference position information, and the corresponding feature in the current feature map is determined according to the feature position information.

[0116] For example, the reference position information of target 1 corresponds to feature positions 1, 2, 3, and 4 in the 4-way current feature map, respectively, and the corresponding features 1, 2, 3, and 4 are obtained from the four feature positions.

[0117] As a variation, for each target reference position, the position information in the current frame corresponding to the reference position information is obtained according to the projection matrix of the camera and the reference position information, and the corresponding feature is determined according to the position information.

[0118] In step 2044, feature fusion is performed for each target to obtain the fusion feature of each target.

[0119] In this step, for the features of the same target, the features of the same target in each current feature map are fused to obtain first fused features, for example, the first features 1, 2, 3, and 4 of the target 1 are fused to obtain the first fused features of the target 1.

[0120] Due to the differences in the camera shooting angle and shooting position, the positions of the same target in the current frame collected by different cameras are different, and the targets corresponding to the same pixel position in the current frame are different. Based on this, the features of other targets except the target in each feature map can also be fused to obtain second fused features, for example, the features of other targets except the target 1 are fused, that is, the features of the target 2, the features of the target 3, and the like are fused. In this way, not only the redundant target information can be removed, but also the expected target information can be enhanced.

[0121] The first fused features and the second fused features are fused to obtain the fused features of the target.

[0122] The above fusion can include at least one of adding and splicing the feature vectors, wherein the adding can be weighted average adding.

[0123] In step 205, based on the projection feature point information, the fused features are searched to obtain the target detection result of the current frame group under the same perspective.

[0124] As an example, the fused features of all targets and the projection feature point information are analyzed through a machine learning model to obtain the target detection result in the BEV perspective, that is, the target detection result in the current frame BEV visual map,

[0125] When the fused features are 3D features, the target detection result includes three-dimensional position information, three-dimensional size information, and confidence, and when the fused features are 2D features, the target detection result includes two-dimensional position information, two-dimensional size information, and confidence.

[0126] In step 206, according to a confidence threshold, the target detection result is filtered to retain valid target detection results,

[0127] In this step, as an example, the target detection result with a confidence less than the confidence threshold is removed to obtain the valid target detection result,

[0128] In step 207, the retained valid target detection result is added to an initial target feature set for target searching of a next frame group,

[0129] The initial target feature set of the next frame group includes an initial target detection vector for target searching.

[0130] For example, if the current frame group has m valid target detection results, then the number of target detection vectors in the next frame group is: the initial n target detection vectors + the historical m valid target detection results. In this case, the current frame group is the historical frame group relative to the next frame group, and the m valid target detection results become the historical valid target detection results.

[0131] Step 208: Determine whether the valid target detection results of the current frame group are related to the valid target detection results of the previous frame group.

[0132] If a valid target detection result comes from the initial target detection, it means that the valid target detection result is newly detected, and a new tracking identifier (ID) is assigned to the valid target detection result;

[0133] If the valid target detection result comes from the valid target detection result added in the previous frame group, it means that the valid target detection result was detected in both the previous frame group (a historical frame group relative to the current frame group) and the current frame group. In this case, the tracking ID of the valid target detection result remains unchanged, and the tracking ID of the previous frame group of the valid target detection result is used.

[0134] The above steps 207 and 208 have no strict order and can be executed in parallel. It should be understood that since the target detection is performed from the same perspective, the current frame group, the next frame group, and the previous frame group mentioned in steps 207 and 208 should be understood as a collection of image frames from each path that are simultaneous, rather than a single image frame from a certain path.

[0135] Step 209: Extract the next frame image from each stream, return to step 203, and continue until all images in the video stream have been processed.

[0136] Step 210: Output the valid target detection results with the same tracking ID, obtain the trajectory data of the valid target detection results in the BEV visual map, and thus obtain the tracking results.

[0137] The process of obtaining target detection results through machine learning models in steps 2042 and 205 above can be understood as the process of querying or searching for a set target in the visual map from the BEV perspective, i.e., the target query process, in order to perform target detection.

[0138] To facilitate understanding of steps 2042 to 207 in the above process, please refer to... Figure 5 As shown, Figure 5 This is a schematic diagram of the target detection process. In the diagram, boxes of different gray levels represent different targets, c li c represents reference location information in a visual map. lmiThe reference position information is projected into the position information in each current image frame, and in the dashed box, target 1 is fused with other target information and the information of the target itself in each current image frame.

[0139] Referring to Figure 6 as shown, Figure 6 is a schematic diagram of a multi-camera target detection and tracking device according to an embodiment of the present application. The device comprises,

[0140] The first acquisition module is configured to acquire at least two video stream image data, wherein each video stream image data comprises at least image data of an overlapping region,

[0141] The second acquisition module is configured to acquire visual map information of a position of the overlapping region in the video stream image data,

[0142] The target detection and tracking module is configured to convert feature point information extracted from each image frame of each video stream image data to a visual map, so as to fuse the feature point information of each image frame to the same perspective, and obtain fusion information under the same perspective, wherein each image frame has simultaneity,

[0143] Based on the fusion information, target detection and tracking are performed.

[0144] Wherein,

[0145] The target detection and tracking module is configured to:

[0146] For each image frame in each image frame:

[0147] Feature extraction is performed on the image frame respectively, to obtain feature point information and / or a feature map of the image frame,

[0148] The projection matrix of the camera from which the image frame is derived is used to project the feature points of the image frame to the visual map, to obtain projection feature point information of the image frame under the same perspective,

[0149] The projection feature point information of all image frames in each image frame is determined as the fusion information,

[0150] Wherein,

[0151] The projection matrix is used to represent the projection matrix of the mapping relationship between the pixel points in the camera image and the spatial points in the visual map;

[0152] Based on the projection feature point information of all image frames in each image frame, target detection is performed to obtain target detection results under the same perspective corresponding to all image frames in each image frame, and the current detection results are determined as the target detection results of the image frame group of each image frame,

[0153] From the target detection results of each image frame group, target trajectory data is acquired,

[0154] The target detection result of each image frame group is a target detection result of a same view angle corresponding to all image frames of each different time.

[0155] Referring to Figure 7 as shown, Figure 7 is another schematic diagram of the multi-camera target detection and tracking device of the embodiment of the present application. The device comprises a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the multi-camera target detection and tracking method of the embodiment of the present application.

[0156] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), for example at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0157] The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0158] The embodiment of the present application also provides a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the multi-camera target detection and tracking method of the embodiment of the present application.

[0159] For the device / network side equipment / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0160] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0161] The application has been described herein in relation to particular embodiments, and any modifications and / or improvements, etc., made to the preferred embodiments are intended to fall within the scope of the application.

Claims

1. A multi-camera target detection and tracking method, characterized in that, The method includes: Acquire image data from at least two video streams from different cameras, wherein each video stream image data includes at least image data of the overlapping region. Obtain visual map information of the spatial location corresponding to the video stream image data. Feature point information extracted from each image frame of each video stream is converted into a visual map to fuse the feature point information of each image frame into the same viewpoint, thus obtaining fused information from the same viewpoint. Each image frame in each stream is simultaneous, and the fused information includes: the projected feature point information of all image frames in each stream. Based on the fused information, target detection and tracking are performed. in, The target detection and tracking based on the fused information includes: Based on the projection feature point information of all image frames in each path of image frames, target detection is performed to obtain the target detection result of all image frames in each path of image frames under the same viewpoint. This target detection result is then determined as the target detection result of the image frame group of each path of image frames. Obtain target trajectory data from the target detection results of each image frame group; The target detection based on the projection feature point information of all image frames in each image frame includes: The projected feature point information and the pre-initialized target detection vector are input into the machine learning model to search the pre-initialized target features and obtain the target's reference position information. By using the projection matrix of the camera from which each image frame originates, the target reference position information is back-projected into each image frame to determine the target reference position's location within the image frame. Based on the positional information of the image frame, the features of the corresponding target are obtained, thus obtaining the target's features in each path. By fusing the features from each path of the target, we obtain the multi-path fused features of the target. The fused features of each target and the projected feature point information are input into a machine learning model to search the fused features and obtain the target detection results.

2. The multi-camera target detection and tracking method as described in claim 1, characterized in that, The step of converting feature point information extracted from each image frame of each video stream image data into a visual map includes: For each image frame in each path: Feature extraction is performed on each image frame to obtain feature point information and / or feature maps for that image frame. Using the projection matrix of the camera from which the image frame originates, the feature points of the image frame are projected onto the visual map to obtain the projected feature point information of the image frame from the same viewpoint. The projection feature point information of all image frames in each image frame is determined as the fusion information. in, The projection matrix is ​​used to characterize the mapping relationship between pixels in the camera image and spatial points in the visual map; the target detection results of each image frame group are: the target detection results of all image frames of each path at different times under the same viewpoint.

3. The multi-camera target detection and tracking method as described in claim 2, characterized in that, The step of obtaining target trajectory data from the target detection results of each image frame group includes: Each target detection result in the intersection of the target detection results of the current image frame group and the target detection results of the historical image frame groups is labeled. From the target detection results labeled in each image frame group, obtain the target trajectory data in the world coordinate system.

4. The multi-camera target detection and tracking method as described in claim 3, characterized in that, The target detection results include: global position information from the same viewpoint, target size, and confidence level.

5. The multi-camera target detection and tracking method as described in claim 4, characterized in that, The target detection vector includes target feature vectors of two or more targets. The step of back-projecting the target reference position information onto each image frame using the projection matrix of the camera from which each image frame originates includes: By using the projection matrix of the camera from which each video stream image originates, the reference position information of each target is back-projected onto the feature map corresponding to each image frame, in order to determine the position information of the reference position of each target in the feature map. The step of obtaining the features of the corresponding target based on the position information of the image frame, and obtaining the target's features in each path, includes: Based on the location information in the feature maps of each target, the features corresponding to each target are obtained.

6. The multi-camera target detection and tracking method as described in claim 4, characterized in that, The fusion of each feature of the target yields the fused features of the target, including: For each target, feature fusion is performed separately to obtain the fused features of each target.

7. The multi-camera target detection and tracking method as described in claim 6, characterized in that, The step of performing feature fusion for each target includes: For each objective: Based on each feature map, the features of the target in each feature map are fused to obtain the first fused feature. Based on each feature map, features of other targets besides the same target are fused from each feature map to obtain the second fused feature. The first fusion feature and the second fusion feature are fused together to obtain the fusion feature of the target; The method further includes: The target detection results of the current image frame group are filtered according to the set confidence threshold to obtain the valid target detection results. The effective target detection results are added to the initial target feature set of the next image frame group.

8. The multi-camera target detection and tracking method as described in claim 7, characterized in that, The step of marking each target detection result in the intersection of the target detection results of the current image frame group and the target detection results of the historical image frame groups includes: If the valid target detection result of the current image frame group comes from the initialized target features, then assign a new tracking label to the valid target detection result; If the valid target detection result of the current image frame group comes from the valid target detection result added in the previous image frame group, then the tracking identifier of the valid target detection result added in the previous image frame group shall be used. The step of obtaining target trajectory data in the world coordinate system from the target detection results marked in each image frame group includes: The target detection results with the same tracking identifier from the target detection results marked in each image frame group are identified as the target trajectory data of that target detection result; The visual map is a bird's-eye view map, and the same viewpoint is a bird's-eye view.

9. A multi-camera target detection and tracking device, characterized in that, The device includes: The first acquisition module is used to acquire at least two video stream image data from different cameras, wherein each video stream image data includes at least image data of the overlapping region. The second acquisition module is used to acquire visual map information of the spatial location corresponding to the video stream image data. The target detection and tracking module extracts feature point information from each image frame of each video stream and converts it into a visual map. This allows the feature point information from each image frame to be fused together under the same viewpoint, resulting in fused information from the same viewpoint. Each image frame within each video stream is simultaneous. Based on the fused information, target detection and tracking are performed; in, The target detection and tracking based on the fused information includes: Based on the projection feature point information of all image frames in each path of image frames, target detection is performed to obtain the target detection result of all image frames in each path of image frames under the same viewpoint. This target detection result is then determined as the target detection result of the image frame group of each path of image frames. Obtain target trajectory data from the target detection results of each image frame group; The target detection based on the projection feature point information of all image frames in each image frame includes: The projected feature point information and the pre-initialized target detection vector are input into the machine learning model to search the pre-initialized target features and obtain the target's reference position information. By using the projection matrix of the camera from which each image frame originates, the target reference position information is back-projected into each image frame to determine the target reference position's location within the image frame. Based on the positional information of the image frame, the features of the corresponding target are obtained, thus obtaining the target's features in each path. By fusing the features from each path of the target, we obtain the multi-path fused features of the target. The fused features of each target and the projected feature point information are input into a machine learning model to search the fused features and obtain the target detection results.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-camera target detection and tracking method as described in any one of claims 1 to 8.

Images