Target tracking method and device based on Leiyu fusion, storage medium and equipment

Abstract

The invention discloses a target tracking method and device based on thunder-vision fusion, a storage medium and equipment, and the method comprises the steps: obtaining video data of a plurality of video devices and radar data of a plurality of radar devices at a traffic intersection, extracting a target video frame and a target radar frame containing a target object, obtaining target video pixel coordinates and target longitude and latitude information, performing plane coordinate conversion to obtain ground plane coordinates, and obtaining a time synchronization frame data set according to timestamps of the target video frame and the target radar frame, the ground plane video coordinates and the ground plane radar coordinates; and the target object coordinates of the target video frame and the target object coordinates of the target radar frame at the same moment are fused to obtain a cross-device thunder-sight fusion result. And finally, matching the cross-device fusion coordinates of each target object in the cross-device thundersight fusion results at different moments to the corresponding trajectory trackers to obtain the tracking trajectory of each target object so as to realize accurate and efficient multi-sensor target fusion.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a target tracking method, device, storage medium and equipment based on radar and vision fusion. Background Art

[0002] In intelligent transportation application scenarios, monocular cameras and millimeter-wave radars are commonly used sensors for target detection. Monocular cameras can obtain target texture information, but are easily affected by lighting and weather, and have a short perception distance. Millimeter-wave radars can measure the speed and spatial position of moving targets and have good adaptability to bad weather such as rain and fog, but cannot perceive stationary targets. Most current cross-scenario target tracking algorithms only use video data, which has the problem of low target detection accuracy and inability to adapt to complex scenarios such as bad weather. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a target tracking method, device, storage medium and equipment based on radar and vision fusion, which can improve the detection accuracy of the target object.

[0004] In a first aspect, the present application provides a target tracking method based on radar-visual fusion, the method comprising:

[0005] Acquire video data of multiple video devices and radar data of multiple radar devices at a traffic intersection, each of the video devices corresponding to multiple frames of video data, and each of the radar devices corresponding to multiple frames of radar data;

[0006] Extracting a target video frame containing a target object from the multiple frames of video data, and extracting target video pixel coordinates corresponding to each target object from each of the target video frames; extracting a target radar frame containing the target object from the multiple frames of radar data, and extracting target longitude and latitude information corresponding to each of the target objects from each of the target radar frames;

[0007] The target video pixel coordinates and the target longitude and latitude information are converted into geodetic plane video coordinates and geodetic plane radar coordinates respectively, and a time synchronization frame data set is obtained according to the timestamps of the target video frame and the target radar frame and the geodetic plane video coordinates and the geodetic plane radar coordinates, wherein the time synchronization frame data set includes synchronized target video frames with a time synchronization correspondence relationship of different video devices and synchronized target radar frames with a time synchronization correspondence relationship of different radar devices, and the synchronized target video frames and the synchronized target radar frames also have a time synchronization correspondence relationship;

[0008] The target objects in the synchronized target video frames of different video devices at the same time are fused to obtain a fused single-source target video frame result, the target objects in the synchronized target radar frames of different radar devices at the same time are fused to obtain a fused single-source target radar frame result, and the single-source target video frame result and the single-source target radar frame result at the same time are fused to obtain a cross-device radar vision fusion result;

[0009] The tracking trajectory of each target object is obtained according to the cross-device fusion coordinates of each target object in the cross-device radar vision fusion results at different times.

[0010] According to the target tracking method based on radar-vision fusion of the present application, target video frames containing target objects from different video devices and target radar frames containing target objects from different radar devices are obtained, and the target video pixel coordinates corresponding to the target objects in the target video frames and the target longitude and latitude information of the target objects in the target radar frames are respectively converted into geodetic plane video coordinates and geodetic plane radar coordinates in the same plane coordinate system, and then the target video frames of different video devices at the same moment are time-synchronized, the target radar frames of different radar devices at the same moment are time-synchronized, and the target video frames and target radar frames at the same moment are time-synchronized to obtain a time-synchronized frame data set with a time synchronization correspondence relationship, and the target objects in the synchronized target video frames of different video devices at the same moment are further fused to obtain a fused The single-source target video frame result after fusion is obtained, the target objects in the synchronized radar frames of different radar devices at the same time are fused to obtain the fused single-source target radar frame result, and then the single-source target video frame result and the single-source target radar frame result at the same time are fused to obtain the cross-device radar vision fusion result, and finally the tracking trajectory of each target object is obtained according to the cross-device fusion coordinates of each target object in the cross-device radar vision fusion result at different times. This method can realize accurate and efficient fusion of target objects of different sensors, and because of the fusion of the complementary advantages of video data and radar data, it can not only improve the detection accuracy of target objects, but also adapt to complex scenes such as bad weather, and this method can be adapted to the tracking of multiple target objects at the same time, can be applied to large-scale scenes such as intersections, and realize adaptive tracking of target objects.

[0011] In a second aspect, the present application provides a target tracking device based on radar-visual fusion, the device comprising:

[0012] A radar data acquisition module, used to acquire video data of multiple video devices and radar data of multiple radar devices at a traffic intersection, each of the video devices corresponds to multiple frames of video data, and each of the radar devices corresponds to multiple frames of radar data;

[0013] A target coordinate extraction module is used to extract a target video frame containing a target object from the multiple frames of video data, and extract target video pixel coordinates corresponding to each target object from each target video frame, extract a target radar frame containing the target object from the multiple frames of radar data, and extract target longitude and latitude information corresponding to each target object from each target radar frame;

[0014] A conversion synchronization module, used for converting the target video pixel coordinates and the target longitude and latitude information into geodetic plane video coordinates and geodetic plane radar coordinates respectively, and obtaining a time synchronization frame data set according to the timestamps of the target video frame and the target radar frame and the geodetic plane video coordinates and the geodetic plane radar coordinates, wherein the time synchronization frame data set includes synchronized target video frames with a time synchronization correspondence relationship of different video devices and synchronized target radar frames with a time synchronization correspondence relationship of different radar devices, and the synchronized target video frames and the synchronized target radar frames also have a time synchronization correspondence relationship;

[0015] A radar vision fusion module is used to fuse the target objects in the synchronized target video frames of different video devices at the same time to obtain a fused single-source target video frame result, fuse the target objects in the synchronized target radar frames of different radar devices at the same time to obtain a fused single-source target radar frame result, and fuse the single-source target video frame result and the single-source target radar frame result at the same time to obtain a cross-device radar vision fusion result;

[0016] The trajectory tracking module is used to obtain the tracking trajectory of each target object according to the cross-device fusion coordinates of each target object in the cross-device radar vision fusion results at different times.

[0017] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the target tracking method based on radar-vision fusion as described in the first aspect above is implemented.

[0018] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the target tracking method based on radar-vision fusion as described in the first aspect above is implemented.

[0019] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the target tracking method based on radar-vision fusion as described in the first aspect.

[0020] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the target tracking method based on radar-vision fusion as described in the first aspect above.

[0021] The above one or more technical solutions in the embodiments of the present application have at least the following technical effects:

[0022] According to the target tracking method, device, storage medium and equipment based on radar-vision fusion of the present application, target video frames containing target objects from different video devices and target radar frames containing target objects from different radar devices are obtained, and the target video pixel coordinates corresponding to the target objects in the target video frames and the target longitude and latitude information of the target objects in the target radar frames are respectively converted into geodetic plane video coordinates and geodetic plane radar coordinates in the same plane coordinate system, and then the target video frames of different video devices at the same time are time-synchronized, the target radar frames of different radar devices at the same time are time-synchronized, and the target video frames and target radar frames at the same time are time-synchronized to obtain a time-synchronized frame data set with a time synchronization correspondence relationship, and the target objects in the synchronized target video frames of different video devices at the same time are further fused to obtain a fused Single-source target video frame results, the target objects in the synchronous radar frames of different radar devices at the same time are fused to obtain the fused single-source target radar frame results, and then the single-source target video frame results and the single-source target radar frame results at the same time are fused to obtain the cross-device radar vision fusion results, and finally the tracking trajectory of each target object is obtained according to the cross-device fusion coordinates of each target object in the cross-device radar vision fusion results at different times. This method and device can realize accurate and efficient fusion of target objects of different sensors, and because of the complementary advantages of the fusion of video data and radar data, it can not only improve the detection accuracy of the target object, but also adapt to complex scenes such as bad weather, and this method can be adapted to the tracking of multiple target objects at the same time, can be applied to large-scale scenes such as intersections, has good scene versatility, and realizes adaptive tracking of target objects.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0025] Figure 1 It is a flowchart of a target tracking method based on radar-visual fusion provided in an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of the structure of a target tracking device based on radar and vision fusion provided in an embodiment of the present application;

[0027] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] In conjunction with the accompanying drawings, the target tracking method based on radar and vision fusion, the target tracking device based on radar and vision fusion, the electronic device and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0031] Among them, the target tracking method based on radar and vision fusion can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0032] The target tracking method based on radar and vision fusion provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the target tracking method based on radar and vision fusion. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, wearable devices, etc. The target tracking method based on radar and vision fusion provided in the embodiment of the present application is described below by taking the electronic device as the execution subject as an example.

[0033] like Figure 1 As shown, the target tracking method based on radar-vision fusion includes the following steps:

[0034] S100, obtaining video data of a plurality of video devices and radar data of a plurality of radar devices at a traffic intersection, each of the video devices corresponding to a plurality of frames of video data, and each of the radar devices corresponding to a plurality of frames of radar data.

[0035] In order to adapt to the application of large-scale scenarios such as traffic intersections, multiple video devices and radar devices are set up at the intersection. Each video device captures video data of a certain period of time to obtain multiple frames of video data, and each radar device captures radar data of a certain period of time to obtain multiple frames of radar data.

[0036] S200, extracting a target video frame containing a target object from the multiple frames of video data, and extracting target video pixel coordinates corresponding to each target object from each of the target video frames, extracting a target radar frame containing the target object from the multiple frames of radar data, and extracting target longitude and latitude information corresponding to each of the target objects from each of the target radar frames.

[0037] The angle ranges of images captured by multiple video devices and radar devices may be different. Some video data and radar data do not contain the target object. Therefore, only target video frames containing target objects and target radar frames containing target objects are selected to reduce the amount of calculation during subsequent fusion.

[0038] In some embodiments, extracting the target video pixel coordinates corresponding to each target object from each target video frame includes: extracting each target object from each target video frame and using the bottom midpoint pixel coordinates of each target object as the target video pixel coordinates.

[0039] In some embodiments, pixel coordinates of other positions of the target object may also be selected as the target video pixel coordinates, such as the centroid pixel coordinates of the target object, the top midpoint pixel coordinates, etc. However, the bottom midpoint pixel coordinates are closest to the ground plane and will not produce height errors, and the coordinate conversion is more accurate. Therefore, in order to make the coordinate conversion more accurate, only the bottom midpoint pixel coordinates of the target object are selected as the target video pixel coordinates. The target objects include motor vehicles, pedestrians, non-motor vehicles, etc.

[0040] In some embodiments, the set of target video pixel coordinates and ground plane video coordinates may be represented as: in, Indicates the bottom midpoint pixel coordinates of the target object in the target video frame under each video device, that is, the target video pixel coordinates. Represents the geodetic plane coordinates of the bottom midpoint of the target object in the target video frame under each video device, that is, the geodetic plane video coordinates, Indicates the number of video devices. Indicates the number of target video frames for each video device. Indicates the number of target objects in each target video frame under each video device. Indicates the category of the target object in each target video frame.

[0041] In some embodiments, a set of geodetic plane radar coordinates may be represented as: in, Represents the geodetic plane coordinates of the midpoint of the bottom edge of the target object in the target radar frame under each radar device, that is, the geodetic plane radar coordinates, represents the number of radar devices, Indicates the number of target radar frames for each radar device, Indicates the number of target objects in each target radar frame under each radar device. Indicates the category of the target object in each target radar frame.

[0042] It should be noted that the latitude and longitude of the bottom midpoint position does not exist in the target radar frame, so the latitude and longitude information can be used directly.

[0043] S300, converting the target video pixel coordinates and the target longitude and latitude information into geodetic plane video coordinates and geodetic plane radar coordinates respectively, and obtaining a time synchronization frame data set according to the timestamps of the target video frame and the target radar frame and the geodetic plane video coordinates and the geodetic plane radar coordinates, wherein the time synchronization frame data set includes synchronized target video frames having a time synchronization correspondence relationship of different video devices and synchronized target radar frames having a time synchronization correspondence relationship of different radar devices, and there is also a time synchronization correspondence relationship between the synchronized target video frames and the synchronized target radar frames.

[0044] In order to fuse the subsequent video pixel coordinates and radar longitude and latitude information, it is necessary to convert the video pixel coordinates and radar longitude and latitude information into the same coordinate system, and the geodetic plane coordinate system is a system that can be realized by both at the same time. Therefore, the target video pixel coordinates are converted into geodetic plane video coordinates, and the target longitude and latitude information is converted into geodetic plane radar coordinates. Among them, in the process of converting the target video pixel coordinates into geodetic plane video coordinates, the mapping relationship between the video data and the geodetic plane coordinate system can be obtained through the video data and the high-precision map data, and then the target video pixel coordinates are converted into geodetic plane video coordinates according to the mapping relationship. Specifically, the road arrow mark can be selected in the video frame and the high-precision map, and the homography matrix of the plane where the video frame road is located and the plane where the high-precision map is located is calculated based on the road arrow mark to obtain the mapping relationship between the video road plane and the geodetic plane coordinate system. The matrix calculation method for obtaining the mapping relationship here can be implemented using existing technology, which will not be repeated here.

[0045] In some embodiments, the step of obtaining the time-synchronized frame data set according to the timestamps of the target video frame and the target radar frame and the geodetic plane video coordinates and the geodetic plane radar coordinates in step S300 can be implemented in the following manner:

[0046] An initial radar matching frame within a preset range corresponding to the target video frame at the same moment is determined according to the timestamps of the target video frame and the target radar frame.

[0047] The Euclidean distance between each target object in the target video frame at the same time and all the target objects in the initial radar matching frame is calculated according to the geodetic plane video coordinates and the geodetic plane radar coordinates, and the initial radar matching frame corresponding to each target object when the Euclidean distance is minimum is used as the radar matching frame at the corresponding time of the target video frame.

[0048] The target video frames at all times and their corresponding radar matching frames are used as the time synchronization frame data set.

[0049] It should be noted that this implementation is to achieve time synchronization of target video frames and target radar frames at different times. It is understandable that time synchronization can be achieved between target video frames only through timestamps, and target radar frames can also be achieved through their respective times. However, due to the difference in frame rates between target video frames and target radar frames, it is not possible to determine whether the target video frames and target radar frames are time synchronized using only timestamps.

[0050] Specifically, the target video frame and the target radar frame at the same time are determined according to the timestamps of the target video frame and the target radar frame, and then the target radar frame set of different radar devices having the same timestamp as the target video frame at the same time is determined according to the timestamp, which is recorded as In order to overcome the problem of different frame rates, a target radar frame that is within a preset range of the target radar frame set is selected and merged with the target radar frame set to obtain the initial radar matching frame, which is recorded as (m k -r,m k +r), the Euclidean distance is calculated by the geodesic video coordinates of each target object in the target video frame at the same time and the geodesic radar coordinates of each target object in the initial radar matching frame, and the initial radar matching frame corresponding to each target object at the minimum Euclidean distance is used as the radar matching frame at the corresponding time of the target video frame, that is, by the formula Obtain, thereby achieving time synchronization of the target video frame and the target radar frame, and obtaining the target video frame at all times and its corresponding radar matching frame. The set of target video frames at all times is recorded as:

[0051]

[0052] The target radar frame set at all times is recorded as:

[0053]

[0054] Among them, N m , They respectively represent the number of frames after time synchronization (including the number of target video frames and the number of target radar frames), the number of target objects corresponding to each target video frame, and the number of target objects corresponding to each target radar frame. The superscript m of the coordinates and target object categories represents the results after time synchronization.

[0055] S400, fuse the target objects in the synchronized target video frames of different video devices at the same time to obtain a fused single-source target video frame result, fuse the target objects in the synchronized target radar frames of different radar devices at the same time to obtain a fused single-source target radar frame result, and fuse the single-source target video frame result and the single-source target radar frame result at the same time to obtain a cross-device radar vision fusion result.

[0056] It can be understood that, based on step S300, a time-synchronized synchronization target video frame is obtained through the timestamp, and then the target objects in the synchronization target video frames at the same time can be fused to obtain a fused single-source target video frame result. Specifically, in some embodiments, the single-source target video frame result can be obtained by:

[0057] At a certain moment, a reference video device is selected from the video devices, and the remaining video devices are used as comparison video devices.

[0058] The target objects in the comparison video device are aligned with the corresponding target objects in the reference video device using the geoplane video coordinates of the reference video device and the geoplane video coordinates of the comparison video device, and the geoplane video coordinates of each target object in each video device after alignment are used as alignment video coordinates.

[0059] The mean of all the aligned video coordinates of each of the target objects is calculated to obtain the fused video pixel coordinates of each of the target objects.

[0060] The single-source target video frame result at the certain moment is obtained according to the fused video pixel coordinates of each target object.

[0061] It should be noted that, since there can be multiple synchronization target video frames at the same time after time synchronization, and a video device can only collect one frame of video data at the same time, each synchronization target video frame corresponds to a video device, so the target objects in multiple synchronization target video frames are fused, which is equivalent to fusion of the target objects in the target video frames collected by multiple video devices at the same time. At a certain moment, a reference video device is first selected from multiple video devices. In some embodiments, the video device with the largest number of target objects is selected as the reference video device. If there are multiple video devices with the largest number of target objects, the video device with a smaller device number is selected as the reference video device. By analogy, if the number of target objects in all video devices is the same, the video device with device number 1 is selected as the reference video device, and the other video devices are used as comparison video devices.

[0062] After selecting the reference video device and the comparison video device, the target objects in the comparison video device are aligned with the corresponding target objects in the reference video device using the geodesic plane video coordinates of the reference video device and the geodesic plane video coordinates of the comparison video device, and the geodesic plane video coordinates of the aligned target objects are used as the alignment video coordinates. In some embodiments, when aligning the target objects in the comparison video device with the corresponding target objects in the reference video device, the formula Calculate the Euclidean distance between each target object in the comparison video device and the corresponding target object in the reference video device, where represents the ground plane video coordinates of the target object in the reference video device, Represents the geodesic plane video coordinates of the target object in the comparison video device, and constructs a target object matching cost table between the comparison video device and the reference video device. The size of the cost table is According to the above Euclidean distance formula, the column index j corresponding to the minimum cost value of each row i is calculated. If i≠j, the id of the target object in the comparison video device is exchanged with the id of the corresponding target object in the reference video device. Otherwise, no processing is performed, thereby achieving the alignment of the target object in the comparison video device with the corresponding target object in the reference video device. For example, if the 6 target objects of the second video device (comparison video device) are aligned with the 5 target objects of the first video (reference video device), if the first target object of the first video device is closest to the third target object of the second video device, then the id of the third target object in the second video device needs to be modified to 1.

[0063] The geodesic plane video coordinates of each target object after alignment are used as the aligned video coordinates, and the mean of all aligned video coordinates of each target object is calculated to obtain the fused video pixel coordinates of each target object. Then, the fused video pixel coordinates of each target object are fused into the same video frame to obtain the single-source target video frame result at a certain moment, thus realizing the single-source fusion of single-source video frames. The set of single-source video frames at different moments is recorded as: Among them, c represents the result after single-source fusion, V subscript, coordinate superscript, and target object category superscript s also represent the result after single-source fusion. Different symbols are used here only for distinction.

[0064] In some embodiments, in step S400, the step of fusing the target objects in the synchronous radar frames of different radar devices at the same time to obtain the fused single-source target radar frame result specifically includes:

[0065] At a certain moment, a reference radar device is selected from the radar devices, and the remaining radar devices are used as comparison radar devices.

[0066] The target objects in the comparison radar device are aligned with the corresponding target objects in the reference radar device using the geodesic plane radar coordinates of the reference radar device and the geodesic plane radar coordinates of the comparison radar device, and the geodesic plane radar coordinates of each target object in each radar device after alignment are used as alignment radar coordinates.

[0067] Calculating the mean of all the aligned radar coordinates of each target object to obtain the fused radar coordinates of each target object;

[0068] The single-source target radar frame result at the certain moment is obtained according to the fused radar coordinates of each target object.

[0069] Similarly, in some embodiments, target objects in synchronized radar frames of different radar devices at the same time are fused to obtain a fused single-source target radar frame result, which can also be obtained in a manner similar to a single-source target video frame result.

[0070] At the same time, a radar device can only collect one frame of radar data. Each synchronous target radar frame corresponds to a radar device. Therefore, fusing the target objects in multiple synchronous target radar frames is equivalent to fusing the target objects in the target radar frames collected by multiple radar devices at the same time. At a certain moment, a reference radar device is first selected from multiple radar devices. In some implementations, the radar device with the largest number of target objects is selected as the reference radar device. If there are multiple radar devices with the largest number of target objects, the radar device with a smaller device number is selected as the reference radar device. By analogy, if the number of target objects in all radar devices is the same, the radar device with a device number of 1 is selected as the reference radar device, and the other radar devices are used as comparison radar devices.

[0071] After selecting the reference radar device and the comparison radar device, the target objects in the comparison radar device are aligned with the corresponding target objects in the reference radar device using the geodesic radar coordinates of the reference radar device and the geodesic radar coordinates of the comparison radar device, and the geodesic radar coordinates of the target objects in the radar devices after alignment are used as the alignment radar coordinates. In some embodiments, when aligning the target objects in the comparison radar device with the corresponding target objects in the reference radar device, the formula The Euclidean distance between each target object in the comparison radar device and the corresponding target object in the reference radar device is calculated, where represents the ground plane radar coordinates of the target object in the reference radar device, Represents the geodesic radar coordinates of the target object in the comparison radar device, and constructs a target object matching cost table between the comparison radar device and the reference radar device. The size of the cost table is According to the above Euclidean distance formula, the column index j corresponding to the minimum cost value of each row i is calculated. If i≠j, the id of the target object in the comparison radar device is exchanged with the id of the corresponding target object in the reference radar device. Otherwise, no processing is performed, thereby realizing the alignment of the target object in the comparison radar device with the corresponding target object in the reference radar device. For example, if the 6 target objects of the second radar device (comparison radar device) are aligned with the 5 target objects of the first radar (reference radar device), if the first target object of the first radar device is closest to the third target object of the second radar device, then the id of the third target object in the second radar device needs to be modified to 1.

[0072] The geodesic radar coordinates of each target object after alignment are used as the aligned radar coordinates, and the mean of all aligned radar coordinates of each target object is calculated to obtain the fused radar coordinates of each target object. Then, the fused radar coordinates of each target object are fused into the same radar frame to obtain the single-source target radar frame result at a certain moment, thus realizing the single-source fusion of single-source radar frames. The set of single-source radar frames at different moments is recorded as: Among them, c represents the result after single-source fusion, R subscript, coordinate superscript, and target object category superscript s also represent the result after single-source fusion. Different symbols are used here only for distinction.

[0073] In some embodiments, in step S400, the single-source target video frame result and the single-source target radar frame result at the same time are fused to obtain a cross-device radar vision fusion result, which specifically includes:

[0074] At the same time, cross-device fusion matching is performed using the fused video pixel coordinates of each target object in the single-source target video frame result and the fused radar coordinates of each target object in the single-source target radar frame result.

[0075] The successfully matched target radar object and target video object are regarded as the same target object, and the average of the fused radar coordinates of the target radar object and the fused video pixel coordinates of the target video object are calculated as the cross-device fused coordinates of the same target object.

[0076] The cross-device radar vision fusion result is obtained from the cross-device fusion coordinates of the same target object.

[0077] As can be seen from the aforementioned step S300, the target video frame and the target radar frame at the same time have been time synchronized. After obtaining the single-source target video frame result and the single-source target radar frame result, cross-device fusion between the video device and the radar device can be performed. Specifically, the fused video pixel coordinates of each target object in the single-source target video frame result and the fused radar coordinates of each target object in the single-source target radar frame result can be used for cross-device fusion matching, that is, the formula is used. And construct the single-source fusion data matching cost table of the single-source target video frame result and the single-source target radar frame result. The size of the cost table is recorded as Use the above Euclidean distance formula to find the column index j corresponding to the minimum cost value of each row i in the single-source fusion data matching cost table as the target object id in the matching target radar frame r , according to id r Reversely search for the row index corresponding to the minimum cost value as the target object ID in the matching target video frame v , when id v=i, it indicates that the target object of the single-source target video frame result and the single-source target radar frame result is matched successfully, the average of the fused radar coordinates of the target radar object and the fused video pixel coordinates of the target video object is calculated, and the cross-device fusion coordinates (i.e., cross-device radar vision fusion pixel coordinates) are output, and the cross-device fusion coordinates of each successfully matched target radar object and target video object are fused into the same data frame to obtain the cross-device radar vision fusion result, otherwise it indicates that the match fails, and the single-source target video frame result and the single-source target radar frame result are retained. Furthermore, the cross-device radar vision fusion results obtained by cross-device fusion at different times are summarized to obtain a cross-device radar vision fusion result set recorded as: in, It indicates the number of target objects in the cross-device radar fusion result. The subscript f of the coordinates and target object category indicates the result after the cross-device radar fusion of the target object.

[0078] S500, obtaining the tracking trajectory of each target object according to the fusion coordinates of each target object in the cross-device radar vision fusion results at different times.

[0079] Specifically, in some embodiments, step S500 includes: constructing a trajectory tracker for each of the target objects, each of the trajectory trackers storing historical cross-device fusion coordinates of the corresponding target object.

[0080] The cross-device fusion coordinates of each target object in the cross-device radar vision fusion result and the historical cross-device fusion coordinates are used to perform trajectory matching.

[0081] The successfully matched cross-device fusion coordinates are added to the matched trajectory tracker.

[0082] Add the unmatched cross-device fusion coordinates to the newly created trajectory tracker.

[0083] By analogy, the cross-device fused coordinates of each target object at all times are added to the corresponding trajectory tracker to obtain the tracking trajectory of each target object.

[0084] After the cross-device radar vision fusion results at different times are obtained in step S400, the cross-device fusion coordinates of each target object in the cross-device radar vision fusion results and the historical cross-device fusion coordinates are used for trajectory matching. Specifically, the Euclidean distance between the cross-device fusion coordinates of the current target object and the previous historical cross-device fusion coordinates in each trajectory tracker can be calculated to avoid redundant calculations, and a fusion data cost table [n] between each target object in the cross-device radar vision fusion result and each target object in all trajectory trackers can be constructed. f ,n t ], where n f Indicates the number of target objects in each cross-device radar fusion result, nt represents the number of total trajectory trackers, and uses the aforementioned Euclidean distance formula to fusion data cost table [n f ,n t ] find the column index corresponding to the minimum cost value of each row i as the matching trajectory tracker id t , according to the matching trajectory tracker id t Reversely search for the row index corresponding to the minimum cost value as the target object ID of the matching radar fusion f , when id f = i, it indicates that there is a matching trajectory tracker for the target object, and the cross-device fusion coordinates of the target object are added to the corresponding nodes in the matching trajectory tracker. Otherwise, it indicates that the match fails, and a new tracker trajectory is created. At the same time, the cross-device fusion coordinates of the target object are added to the newly created tracker trajectory. Finally, the trajectory numbers and node lists of all target objects are output to complete the tracking of the target object fused by the radar vision. Among them, each trajectory tracker corresponds to a trajectory number. When constructing the trajectory tracker of each target object, the trajectory tracker of each target object is numbered.

[0085] Optionally, in some embodiments, the step of constructing a trajectory tracker for each target object in the above embodiment is not a necessary step and can be pre-constructed, where each target object corresponds to a trajectory tracker, each trajectory tracker corresponds to a trajectory number, and each trajectory tracker stores the historical cross-device fusion coordinates of the corresponding target object. Through the aforementioned matching rules, the cross-device fusion coordinates of each target object at different times are filled in the corresponding nodes of the corresponding trajectory tracker in turn to realize the trajectory tracking of the target object over a period of time.

[0086] According to the target tracking method based on radar-vision fusion provided in the embodiment of the present application, target video frames containing target objects from different video devices and target radar frames containing target objects from different radar devices are obtained, and the target video pixel coordinates corresponding to the target objects in the target video frames and the target longitude and latitude information of the target objects in the target radar frames are respectively converted into geodetic plane video coordinates and geodetic plane radar coordinates in the same geodetic plane coordinate system, and then the target video frames of different video devices at the same time are time-synchronized, the target radar frames of different radar devices at the same time are time-synchronized, and the target video frames and target radar frames at the same time are time-synchronized to obtain a time-synchronized frame data set with a time synchronization correspondence relationship, and the target objects in the synchronized target video frames of different video devices at the same time are further fused. , obtain the fused single-source target video frame result, fuse the target objects in the synchronous radar frames of different radar devices at the same time, obtain the fused single-source target radar frame result, and then fuse the single-source target video frame result and the single-source target radar frame result at the same time to obtain the cross-device radar vision fusion result, and finally obtain the tracking trajectory of each target object according to the cross-device fusion coordinates of each target object in the cross-device radar vision fusion result at different times. This method can realize accurate and efficient target fusion of different sensors, and because of the complementary advantages of the fusion of video data and radar data, it can not only improve the detection accuracy of the target object, but also adapt to complex scenes such as bad weather, and this method can be adapted to the tracking of multiple target objects at the same time, can be applied to large-scale scenes such as intersections, and realize adaptive tracking of target objects.

[0087] The target tracking method based on thunder and vision fusion provided in the embodiment of the present application can be executed by a target tracking device based on thunder and vision fusion. In the embodiment of the present application, the target tracking method based on thunder and vision fusion is executed by a target tracking device based on thunder and vision fusion as an example to illustrate the target tracking device based on thunder and vision fusion provided in the embodiment of the present application.

[0088] The present application also provides a target tracking device based on radar and vision fusion, such as Figure 2 As shown, the target tracking device based on radar and visual fusion includes: a radar and visual data acquisition module, which is used to acquire video data of multiple video devices and radar data of multiple radar devices at a traffic intersection, each of the video devices corresponds to multiple frames of video data, and each of the radar devices corresponds to multiple frames of radar data.

[0089] The target coordinate extraction module is used to extract the target video frames containing the target object from the multiple frames of video data, and extract the target video pixel coordinates corresponding to each target object from each of the target video frames, extract the target radar frames containing the target object from the multiple frames of radar data, and extract the target longitude and latitude information corresponding to each of the target objects from each of the target radar frames.

[0090] A conversion synchronization module is used to convert the target video pixel coordinates and the target longitude and latitude information into geodetic plane video coordinates and geodetic plane radar coordinates respectively, and obtain a time synchronization frame data set according to the timestamps of the target video frame and the target radar frame and the geodetic plane video coordinates and the geodetic plane radar coordinates, wherein the time synchronization frame data set includes synchronized target video frames with a time synchronization correspondence relationship of different video devices and synchronized target radar frames with a time synchronization correspondence relationship of different radar devices, and the synchronized target video frames and the synchronized target radar frames also have a time synchronization correspondence relationship.

[0091] The radar vision fusion module is used to fuse the target objects in the synchronized target video frames of different video devices at the same time to obtain the fused single-source target video frame result, fuse the target objects in the synchronized target radar frames of different radar devices at the same time to obtain the fused single-source target radar frame result, and fuse the single-source target video frame result and the single-source target radar frame result at the same time to obtain the cross-device radar vision fusion result.

[0092] The trajectory tracking module is used to obtain the tracking trajectory of each target object according to the cross-device fusion coordinates of each target object in the cross-device radar vision fusion results at different times.

[0093] According to the target tracking device based on radar-vision fusion provided by the embodiment of the present application, target video frames containing target objects from different video devices and target radar frames containing target objects from different radar devices are obtained, and the target video pixel coordinates corresponding to the target objects in the target video frames and the target longitude and latitude information of the target objects in the target radar frames are respectively converted into geodetic plane video coordinates and geodetic plane radar coordinates in the same plane coordinate system, and then the target video frames of different video devices at the same time are time-synchronized, the target radar frames of different radar devices at the same time are time-synchronized, and the target video frames and target radar frames at the same time are time-synchronized to obtain a time synchronization frame data set with a time synchronization correspondence relationship, and the target objects in the synchronized target video frames of different video devices at the same time are further fused to obtain The fused single-source target video frame result is used to fuse the target objects in the synchronous radar frames of different radar devices at the same time to obtain the fused single-source target radar frame result, and then the single-source target video frame result and the single-source target radar frame result at the same time are fused to obtain the cross-device radar vision fusion result, and finally the tracking trajectory of each target object is obtained according to the cross-device fusion coordinates of each target object in the cross-device radar vision fusion result at different times. The device can realize accurate and efficient fusion of target objects of different sensors, and because of the complementary advantages of the fusion of video data and radar data, it can not only improve the detection accuracy of the target object, but also adapt to complex scenes such as bad weather, and the method can be adapted to the tracking of multiple target objects at the same time, can be applied to large-scale scenes such as intersections, and realize adaptive tracking of target objects.

[0094] In some embodiments, the target coordinate extraction module is further used to extract each of the target objects from each of the target video frames; and use the bottom midpoint pixel coordinates of each of the target objects as the target video pixel coordinates.

[0095] In some embodiments, the conversion synchronization module is also used to determine the initial radar matching frame within a preset range corresponding to the target video frame at the same time according to the timestamps of the target video frame and the target radar frame; calculate the Euclidean distance between each target object in the target video frame at the same time and all the target objects in the initial radar matching frame according to the geodesic plane video coordinates and the geodesic plane radar coordinates, and use the initial radar matching frame corresponding to each target object when the Euclidean distance is minimum as the radar matching frame at the corresponding time of the target video frame; use the target video frames at all times and their corresponding radar matching frames as the time synchronization frame data set.

[0096] In some embodiments, the radar vision fusion module is also used to select a reference video device from the video devices at a certain moment, and use the remaining video devices as comparison video devices; use the geodesic plane video coordinates of the reference video device and the geodesic plane video coordinates of the comparison video device to align each of the target objects in the comparison video device with the corresponding target objects in the reference video device, and use the geodesic plane video coordinates of each of the target objects in each of the video devices after alignment as the aligned video coordinates; calculate the mean of all the aligned video coordinates of each of the target objects to obtain the fused video pixel coordinates of each of the target objects; and obtain the single-source target video frame result at the certain moment according to the fused video pixel coordinates of each of the target objects.

[0097] In some embodiments, the radar-vision fusion module is also used to select a reference radar device from the radar devices at a certain moment, and use the remaining radar devices as comparison radar devices; align each of the target objects in the comparison radar device with the corresponding target objects in the reference radar device using the geodesic plane radar coordinates of the reference radar device and the geodesic plane radar coordinates of the comparison radar device, and use the geodesic plane radar coordinates of each of the target objects in each of the radar devices after alignment as the alignment radar coordinates; calculate the mean of all the aligned radar coordinates of each target object to obtain the fused radar coordinates of each target object; and obtain the single-source target radar frame result at the certain moment according to the fused radar coordinates of each target object.

[0098] In some embodiments, the radar-vision fusion module is also used to perform cross-device fusion matching at the same time using the fused video pixel coordinates of each target object in the single-source target video frame result and the fused radar coordinates of each target object in the single-source target radar frame result; treating the successfully matched target radar object and target video object as the same target object, and calculating the average of the fused radar coordinates of the target radar object and the fused video pixel coordinates of the target video object as the cross-device fusion coordinates of the same target object; and obtaining the cross-device radar-vision fusion result from the cross-device fusion coordinates of each of the same target objects.

[0099] In some embodiments, the trajectory tracking module is also used to construct a trajectory tracker for each of the target objects, each of the trajectory trackers stores the historical cross-device fusion coordinates of the corresponding target object; trajectory matching is performed using the cross-device fusion coordinates of each target object in the cross-device radar fusion result and the historical cross-device fusion coordinates; the successfully matched cross-device fusion coordinates are added to the matched trajectory tracker; the unsuccessfully matched cross-device fusion coordinates are added to the newly created trajectory tracker; and so on, the cross-device fusion coordinates of each of the target objects at all times are added to the corresponding trajectory tracker to obtain the tracking trajectory of each of the target objects.

[0100] The target tracking device based on radar-visual fusion in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or a device other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0101] The target tracking device based on radar and vision fusion in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0102] The target tracking device based on radar and vision fusion provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0103] In some embodiments, Figure 3As shown, an embodiment of the present application further provides an electronic device 300, including a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the program is executed by the processor 301, each process of the above-mentioned target tracking method embodiment based on radar-vision fusion is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0104] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0105] The embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned target tracking method embodiment based on radar and vision fusion are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0106] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0107] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned target tracking method based on radar and vision fusion when executed by a processor.

[0108] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0109] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned target tracking method embodiment based on radar and vision fusion, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0110] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0111] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0113] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0114] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0115] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A target tracking method based on radar and vision fusion, characterized in that: The method comprises: Acquire video data of multiple video devices and radar data of multiple radar devices at a traffic intersection, each of the video devices corresponding to multiple frames of video data, and each of the radar devices corresponding to multiple frames of radar data; Extracting a target video frame containing a target object from the multiple frames of video data, and extracting target video pixel coordinates corresponding to each target object from each of the target video frames; extracting a target radar frame containing the target object from the multiple frames of radar data, and extracting target longitude and latitude information corresponding to each of the target objects from each of the target radar frames; The target video pixel coordinates and the target longitude and latitude information are converted into geodetic plane video coordinates and geodetic plane radar coordinates respectively, and a time synchronization frame data set is obtained according to the timestamps of the target video frame and the target radar frame and the geodetic plane video coordinates and the geodetic plane radar coordinates, wherein the time synchronization frame data set includes synchronized target video frames with a time synchronization correspondence relationship of different video devices and synchronized target radar frames with a time synchronization correspondence relationship of different radar devices, and the synchronized target video frames and the synchronized target radar frames also have a time synchronization correspondence relationship; The target objects in the synchronized target video frames of different video devices at the same time are fused to obtain a fused single-source target video frame result, the target objects in the synchronized target radar frames of different radar devices at the same time are fused to obtain a fused single-source target radar frame result, and the single-source target video frame result and the single-source target radar frame result at the same time are fused to obtain a cross-device radar vision fusion result; The tracking trajectory of each target object is obtained according to the cross-device fusion coordinates of each target object in the cross-device radar vision fusion results at different times.

2. The method according to claim 1, characterized in that The step of extracting target video pixel coordinates corresponding to each target object from each target video frame includes: Extracting each of the target objects from each of the target video frames; The bottom midpoint pixel coordinates of each target object are used as the target video pixel coordinates.

3. The method according to claim 2, characterized in that The step of obtaining a time-synchronized frame data set according to the timestamps of the target video frame and the target radar frame and the geodetic plane video coordinates and the geodetic plane radar coordinates comprises: Determine an initial radar matching frame within a preset range corresponding to the target video frame at the same time according to the timestamps of the target video frame and the target radar frame; Calculate the Euclidean distance between each target object in the target video frame at the same time and all the target objects in the initial radar matching frame according to the geodetic plane video coordinates and the geodetic plane radar coordinates, and use the initial radar matching frame where each target object corresponding to the minimum Euclidean distance is located as the radar matching frame at the corresponding time of the target video frame; The target video frames at all times and their corresponding radar matching frames are used as the time synchronization frame data set.

4. The method according to claim 3, characterized in that The method of fusing the target objects in the synchronized target video frames of different video devices at the same time to obtain a fused single-source target video frame result includes: At a certain moment, a reference video device is selected from the video devices, and the remaining video devices are used as comparison video devices; Aligning each target object in the comparison video device with the corresponding target object in the reference video device using the geodesic plane video coordinates of the reference video device and the geodesic plane video coordinates of the comparison video device, and using the geodesic plane video coordinates of each target object after alignment as the alignment video coordinates; Calculating the mean of all the aligned video coordinates of each of the target objects to obtain the fused video pixel coordinates of each of the target objects; The single-source target video frame result at the certain moment is obtained according to the fused video pixel coordinates of each target object.

5. The method according to claim 4, characterized in that The step of fusing the target objects in the synchronous radar frames of different radar devices at the same time to obtain a fused single-source target radar frame result includes: At a certain moment, a reference radar device is selected from the radar devices, and the remaining radar devices are used as comparison radar devices; aligning each target object in the comparison radar device with a corresponding target object in the reference radar device using the geodesic plane radar coordinates of the reference radar device and the geodesic plane radar coordinates of the comparison radar device, and using the geodesic plane radar coordinates of each target object after alignment as the alignment radar coordinates; Calculating the mean of all the aligned radar coordinates of each target object to obtain the fused radar coordinates of each target object; The single-source target radar frame result at the certain moment is obtained according to the fused radar coordinates of each target object.

6. The method according to claim 5, characterized in that The fusing the single-source target video frame result and the single-source target radar frame result at the same time to obtain a cross-device radar vision fusion result includes: At the same time, cross-device fusion matching is performed using the fused video pixel coordinates of each target object in the single-source target video frame result and the fused radar coordinates of each target object in the single-source target radar frame result; The successfully matched target radar object and target video object are regarded as the same target object, and the average of the fused radar coordinates of the target radar object and the fused video pixel coordinates of the target video object are calculated as the cross-device fused coordinates of the same target object; The cross-device radar vision fusion result is obtained from the cross-device fusion coordinates of the same target object.

7. The method according to claim 6, characterized in that The obtaining the tracking trajectory of each target object according to the cross-device fusion coordinates of each target object in the cross-device radar vision fusion results at different times includes: Constructing a trajectory tracker for each of the target objects, each of the trajectory trackers storing historical cross-device fusion coordinates of the corresponding target object; Using the cross-device fusion coordinates of each target object in the cross-device radar vision fusion result and the historical cross-device fusion coordinates to perform trajectory matching; Add the successfully matched cross-device fusion coordinates to the matched trajectory tracker; Add the unmatched cross-device fusion coordinates to the newly created trajectory tracker; By analogy, the cross-device fusion coordinates of each target object at all times are added to the corresponding trajectory tracker to obtain the tracking trajectory of each target object.

8. A target tracking device based on radar and vision fusion, characterized in that: The device comprises: A radar data acquisition module, used to acquire video data of multiple video devices and radar data of multiple radar devices at a traffic intersection, each of the video devices corresponds to multiple frames of video data, and each of the radar devices corresponds to multiple frames of radar data; A target coordinate extraction module is used to extract a target video frame containing a target object from the multiple frames of video data, and extract target video pixel coordinates corresponding to each target object from each target video frame, extract a target radar frame containing the target object from the multiple frames of radar data, and extract target longitude and latitude information corresponding to each target object from each target radar frame; A conversion synchronization module, used for converting the target video pixel coordinates and the target longitude and latitude information into geodetic plane video coordinates and geodetic plane radar coordinates respectively, and obtaining a time synchronization frame data set according to the timestamps of the target video frame and the target radar frame and the geodetic plane video coordinates and the geodetic plane radar coordinates, wherein the time synchronization frame data set includes synchronized target video frames with a time synchronization correspondence relationship of different video devices and synchronized target radar frames with a time synchronization correspondence relationship of different radar devices, and the synchronized target video frames and the synchronized target radar frames also have a time synchronization correspondence relationship; A radar vision fusion module is used to fuse the target objects in the synchronized target video frames of different video devices at the same time to obtain a fused single-source target video frame result, fuse the target objects in the synchronized target radar frames of different radar devices at the same time to obtain a fused single-source target radar frame result, and fuse the single-source target video frame result and the single-source target radar frame result at the same time to obtain a cross-device radar vision fusion result; The trajectory tracking module is used to obtain the tracking trajectory of each target object according to the cross-device fusion coordinates of each target object in the cross-device radar vision fusion results at different times.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the target tracking method based on radar and vision fusion as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target tracking method based on radar and vision fusion as described in any one of claims 1 to 7 is implemented.

Images