Target tracking method and device based on multiple sensors, storage medium and vehicle

Abstract

The invention discloses a target tracking method and device based on multiple sensors, a storage medium and a vehicle, and the multiple sensors comprise a camera and a radar, and the method comprises the steps: obtaining the image data of a current frame of a target object through the camera, and obtaining the radar data of the current frame of the target object through the radar; determining a visual target of the current frame according to the image data, and determining a radar target of the current frame according to the radar data; obtaining a prediction target of the current frame of the target object; respectively converting the prediction target, the visual target and the radar target into a world coordinate system; respectively matching the predicted target with a visual target and a radar target in a world coordinate system; and if the predicted target is successfully matched with the visual target or the predicted target is successfully matched with the radar target, tracking the target object by using the coordinates of the visual target and the radar target in the world coordinate system. Therefore, the stability, accuracy and reliability of target tracking are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicles, in particular to a target tracking method and device based on multiple sensors, a storage medium and a vehicle. BACKGROUND

[0002] Target tracking methods mostly rely on a single sensor, such as using only a camera for visual tracking. Such methods are susceptible to factors such as changes in light, target occlusion and complex background, resulting in unstable target detection and easy interruption of tracking. Or only using radar, although radar can provide target distance and speed information, traditional radar tracking algorithms are mainly based on point target models, only extracting limited kinematic features. As the number of targets increases and the complexity of the scene improves, it is difficult for the point target model to distinguish targets with similar motion characteristics. SUMMARY

[0003] The present application aims to at least partially solve one of the technical problems in the related art. To this end, the first object of the present application is to propose a target tracking method based on multiple sensors, which can reduce the loss or false tracking of targets and effectively improve the stability, accuracy and reliability of target tracking.

[0004] The second object of the present application is to propose a computer-readable storage medium.

[0005] The third object of the present application is to propose a target tracking device based on multiple sensors.

[0006] The fourth object of the present application is to propose a vehicle.

[0007] In order to achieve the above-mentioned objects, the first aspect of the present application proposes a target tracking method based on multiple sensors, wherein the multiple sensors include a camera and a radar, and the method comprises: acquiring image data of a target object current frame through the camera, and acquiring radar data of the target object current frame through the radar; determining a visual target of the current frame according to the image data, and determining a radar target of the current frame according to the radar data; acquiring a predicted target of the target object current frame; converting the predicted target, the visual target and the radar target to a world coordinate system respectively; performing target matching between the predicted target and the visual target and the radar target in the world coordinate system; if the predicted target matches the visual target successfully, or the predicted target matches the radar target successfully, tracking the target object using the coordinates of the visual target and the radar target in the world coordinate system.

[0008] According to the multi-sensor-based target tracking method of the embodiment of the present application, the image data of the current frame of the target object is acquired by the camera, and the radar data of the current frame of the target object is acquired by the radar, and the visual target of the current frame is determined according to the image data, and the radar target of the current frame is determined according to the radar data, and the predicted target of the current frame of the target object is acquired, so as to respectively convert the predicted target, the visual target and the radar target to the world coordinate system, and then the predicted target is matched with the visual target and the radar target in the world coordinate system, and if the predicted target is successfully matched with the visual target or the predicted target is successfully matched with the radar target, the target object is tracked by using the coordinates of the visual target and the radar target in the world coordinate system, so as to reduce the situation of losing or incorrectly tracking the target, and effectively improve the stability, accuracy and reliability of the target tracking.

[0009] In addition, the multi-sensor-based target tracking method according to the above-mentioned embodiments of the present application can further include the following additional technical features:

[0010] According to an embodiment of the present application, the predicted target of the current frame of the target object is acquired by acquiring the tracking target of the target object in the previous frame, and predicting the tracking target of the target object in the current frame according to the tracking target of the previous frame to obtain the predicted target of the current frame.

[0011] According to an embodiment of the present application, the tracking target of the target object in the current frame is predicted according to the tracking target of the previous frame, including: acquiring the speed and acceleration of the tracking target of the previous frame; and predicting the tracking target of the target object in the current frame by using a kinematic model according to the speed and the acceleration to obtain the predicted target of the current frame.

[0012] According to an embodiment of the present application, the target object is tracked by using the coordinates of the visual target and the radar target in the world coordinate system, including: acquiring the first coordinate and the first weight coefficient of the visual target in the world coordinate system, and acquiring the second coordinate and the second weight coefficient of the radar target in the world coordinate system; performing weighted calculation and processing on the first coordinate and the second coordinate according to the first weight coefficient and the second weight coefficient to obtain the target object coordinate; and tracking the target object according to the target object coordinate.

[0013] According to an embodiment of the present application, the method further includes: acquiring the environmental information around the camera and the radar; and updating the first weight coefficient and the second weight coefficient according to the environmental information.

[0014] According to one embodiment of the present application, before tracking the target object by using the coordinates of the visual target and the radar target in the world coordinate system, the method further comprises: obtaining speed information and RCS (Radar Cross Section) reflection intensity information of the radar target, and obtaining texture information and color information of the visual target; and performing comprehensive evaluation on the speed information, the RCS reflection intensity information, the texture information and the color information according to a preset speed range, a RCS reflection intensity threshold, preset texture information and a preset color range, so as to filter false targets.

[0015] According to one embodiment of the present application, the method further comprises: obtaining radar point cloud data; and performing motion compensation interpolation processing on the radar point cloud data, so as to realize time synchronization of the radar data and the image data.

[0016] To achieve the above object, the second aspect of the present application provides a computer readable storage medium, which stores a multi-sensor based target tracking program, and the multi-sensor based target tracking program is executed by a processor to realize the multi-sensor based target tracking method of the foregoing embodiments of the present application.

[0017] The computer readable storage medium according to the embodiments of the present application can reduce the situation of missing or false tracking targets by using the multi-sensor based target tracking program, and effectively improve the stability, accuracy and reliability of target tracking.

[0018] To achieve the above object, the third aspect of the present application provides a multi-sensor based target tracking device, characterized in that the multi-sensor comprises a camera and a radar, and the device comprises: a first acquisition module, configured to acquire image data of a current frame of a target object by using the camera, and acquire radar data of the current frame of the target object by using the radar; a determination module, configured to determine a visual target of the current frame according to the image data, and determine a radar target of the current frame according to the radar data; a second acquisition module, configured to acquire a predicted target of the current frame of the target object; a conversion module, configured to convert the predicted target, the visual target and the radar target to a world coordinate system respectively; and a processing module, configured to perform target matching between the predicted target and the visual target and the radar target in the world coordinate system respectively, and track the target object by using the coordinates of the visual target and the radar target in the world coordinate system if the predicted target matches the visual target successfully or the predicted target matches the radar target successfully.

[0019] According to the multi-sensor based target tracking device of the embodiment of the present application, the image data of the current frame of the target object is acquired by the first acquisition module through the camera, and the radar data of the current frame of the target object is acquired by the radar, the visual target of the current frame is determined by the determination module according to the image data, and the radar target of the current frame is determined according to the radar data, and the predicted target of the current frame of the target object is acquired by the second acquisition module, so that the predicted target, the visual target and the radar target are converted to the world coordinate system by the conversion module, and then the predicted target is matched with the visual target and the radar target in the world coordinate system by the processing module, if the predicted target is matched with the visual target successfully, or the predicted target is matched with the radar target successfully, the target object is tracked by using the coordinates of the visual target and the radar target in the world coordinate system, so as to reduce the situation of losing or tracking the target incorrectly, and effectively improve the stability, accuracy and reliability of the target tracking.

[0020] In order to achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present application provides a vehicle comprising the multi-sensor based target tracking device of the aforementioned embodiment of the present application.

[0021] According to the vehicle of the embodiment of the present application, by adopting the multi-sensor based target tracking device of the above-mentioned embodiment of the present application, the situation of losing or tracking the target incorrectly can be reduced, and the stability, accuracy and reliability of the target tracking can be effectively improved.

[0022] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a flow diagram of the multi-sensor based target tracking method according to the embodiment of the present application;

[0024] Figure 2 is a block diagram of the multi-sensor based target tracking device according to the embodiment of the present application;

[0025] Figure 3 is a block diagram of the vehicle according to the embodiment of the present application. DETAILED DESCRIPTION

[0026] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0027] A multi-sensor based target tracking method, a computer readable storage medium, a multi-sensor based target tracking device and a vehicle according to embodiments of the present application are described below with reference to the accompanying drawings.

[0028] Figure 1 is a flowchart of a multi-sensor based target tracking method according to embodiments of the present application.

[0029] Specifically, in some embodiments of the present application, the multi-sensor includes a camera and a radar, such as Figure 1 As shown, the multi-sensor based target tracking method includes:

[0030] S101, obtaining image data of a current frame of a target object through a camera and radar data of the current frame of the target object through a radar.

[0031] Specifically, in this embodiment, a plurality of cameras and a plurality of radars are installed on the vehicle, the target object can be other vehicles or pedestrians, etc., and then the image data of the target object can be collected by the plurality of cameras installed on the vehicle to obtain the image data of the current frame of the target object, and the electromagnetic wave signals emitted by the plurality of radars and the radar signals reflected by the target object are received to obtain the radar data of the current frame of the target object.

[0032] S102, determining a visual target of the current frame according to the image data and a radar target of the current frame according to the radar data.

[0033] Specifically, in this embodiment, the target detection algorithm can be used to process the image data of the current frame of the target object to determine the visual target of the current frame, and the target detection algorithm can recognize different categories of targets (such as vehicles, pedestrians, etc.) in the image and give the position of each target (usually in the form of a bounding box).

[0034] For the radar data of the current frame of the target object obtained, the feature information of the target object such as distance, speed, angle, etc. can be extracted through a signal processing algorithm (such as fast Fourier transform, window function processing, etc.). Then, based on these feature information, a target detection algorithm (such as a clustering-based algorithm, a constant false alarm rate detection algorithm, etc.) is used to determine the radar target of the current frame. When processing the radar signal, noise and interference signals can also be suppressed to improve the reliability of target detection.

[0035] S103, obtaining a predicted target of the current frame of the target object.

[0036] Specifically, in this embodiment, the tracking target of the target object in the previous frame can be obtained from the historical data, and the speed and acceleration of the tracking target in the previous frame are further obtained, so that the tracking target of the target object in the current frame can be predicted according to the speed and acceleration by using a kinematic model. In the prediction process, by substituting the tracking target position, speed and acceleration and other information of the previous frame into the kinematic model, and combining the time interval between the current frame and the previous frame, the position of the tracking target in the current frame, i.e., the predicted target in the current frame, can be calculated.

[0037] S104, respectively, the predicted target, the visual target and the radar target are converted to the world coordinate system.

[0038] Specifically, in this embodiment, a world coordinate system can be established as a unified reference system for all target positions, and the coordinates of the predicted target, the visual target and the radar target are converted from the coordinate systems of the respective sensors to the world coordinate system. For the camera, the internal and external parameters (such as focal length, optical center, rotation matrix, translation vector, etc.) of the camera are obtained through camera calibration, and then the visual target coordinates in the image coordinate system are converted to the world coordinate system by using projection transformation and other methods. For the radar, according to the installation position and attitude of the radar, the conversion relationship (such as rotation matrix and translation vector) between the radar coordinate system and the world coordinate system is determined, and the radar target coordinates in the radar coordinate system are converted to the world coordinate system. At the same time, the coordinates of the predicted target also need to be converted to the world coordinate system according to the motion model and the initial position of the target and other information.

[0039] S105, in the world coordinate system, the predicted target is matched with the visual target and the radar target respectively.

[0040] Specifically, in this embodiment, the first coordinate of the visual target in the world coordinate system and the second coordinate of the radar target in the world coordinate system, and the third coordinate of the predicted target in the world coordinate system can be obtained, so that it can be judged whether the first coordinate is within a preset range of the third coordinate, and whether the second coordinate is within a preset range of the third coordinate.

[0041] S106, if the predicted target is successfully matched with the visual target, or the predicted target is successfully matched with the radar target, the target object is tracked by using the coordinates of the visual target and the radar target in the world coordinate system.

[0042] Specifically, in this embodiment, the first coordinates of the visual target in the world coordinate system can be obtained, and the second coordinates of the radar target in the world coordinate system can be obtained, and the third coordinates of the predicted target in the world coordinate system can be obtained, so that the first coordinates can be matched with the third coordinates, and the second coordinates can be matched with the third coordinates. If the first coordinates are within the preset range of the third coordinates, it is determined that the predicted target and the visual target are matched successfully, and if the second coordinates are within the preset range of the third coordinates, it is determined that the predicted target and the visual target are matched successfully. Further, when the predicted target and the visual target are matched successfully, or the predicted target and the radar target are matched successfully, the first weight coefficient of the visual target is obtained, and the second weight coefficient of the radar target is obtained, and the first coordinates and the second coordinates are weighted and calculated according to the first weight coefficient and the second weight coefficient, so as to obtain the target object coordinates, so as to track the target object according to the target object coordinates.

[0043] Further, in some embodiments of the present application, the predicted target of the target object in the current frame is obtained, including: obtaining the tracking target of the target object in the last frame; predicting the tracking target of the target object in the current frame according to the tracking target of the target object in the last frame to obtain the predicted target in the current frame.

[0044] Specifically, in this embodiment, the tracking target of the target object in the last frame can be obtained from the historical data, and then the speed and acceleration of the tracking target in the last frame can be obtained, so that the tracking target of the target object in the current frame can be predicted according to the speed and acceleration by using the kinematic model. In the prediction process, by substituting the position, speed and acceleration of the tracking target in the last frame into the kinematic model, and combining the time interval between the current frame and the last frame, the position of the tracking target in the current frame, i.e. the predicted target in the current frame, can be calculated.

[0045] Further, in some embodiments of the present application, the target object is tracked by using the coordinates of the visual target and the radar target in the world coordinate system, including: obtaining the first coordinates and the first weight coefficient of the visual target in the world coordinate system, and obtaining the second coordinates and the second weight coefficient of the radar target in the world coordinate system; the first coordinates and the second coordinates are weighted and calculated according to the first weight coefficient and the second weight coefficient to obtain the target object coordinates; the target object is tracked according to the target object coordinates.

[0046] Specifically, in this embodiment, after the visual target is converted to the world coordinate system, a first coordinate of the visual target in the world coordinate system can be obtained, and a first weight coefficient associated with the first coordinate is determined, wherein the first weight coefficient can be determined according to factors such as the confidence of the camera, the environmental lighting condition, the target occlusion condition, etc. For example, in the case of good lighting and no target occlusion, the first weight coefficient can be set to a higher value; while in the case of dim lighting and partial target occlusion, the first weight coefficient is appropriately reduced.

[0047] After the radar target is converted to the world coordinate system, a second coordinate of the radar target in the world coordinate system can be obtained, and a second weight coefficient associated with the second coordinate is determined, wherein the second weight coefficient is determined according to factors such as the signal strength of the radar, the signal-to-noise ratio, the target reflection characteristics, etc. For example, when the radar signal is strong and the signal-to-noise ratio is high, the second weight coefficient can be set to a higher value; on the contrary, if the radar signal is weak and seriously interfered, the second weight coefficient is reduced. Wherein the sum of the first weight coefficient and the second weight coefficient is 1.

[0048] After obtaining the first coordinate, the first weight coefficient, the second coordinate and the second weight coefficient, the first coordinate and the second coordinate are weighted and calculated according to the first weight coefficient and the second weight coefficient, for example, target object coordinate = (first coordinate x first weight coefficient + second coordinate x second weight coefficient), after obtaining the target object coordinate, it is regarded as the position information of the target object at the current time, and is updated to the state variable of the target tracking algorithm, so as to realize the tracking of the target object.

[0049] Further, in some embodiments of the present application, the target tracking method further comprises: obtaining environmental information around the camera and the radar; updating the first weight coefficient and the second weight coefficient according to the environmental information.

[0050] Specifically, in this embodiment, a variety of environmental perception sensors can be equipped on the vehicle to collect surrounding environmental information, such as a rain sensor that can measure rainfall intensity, a lighting sensor that can sense environmental lighting intensity, a sensor for detecting fog concentration, in addition, the type of road (such as expressway, urban road, etc.) and traffic flow information of the vehicle can also be obtained through the vehicle navigation system.

[0051] Further, the first weight coefficient (corresponding to the visual target) and the second weight coefficient (corresponding to the radar target) can be adjusted according to the preset weight adjustment strategy based on the acquired environmental information. For example, in a rainy environment: when the rainfall sensor detects that the rainfall intensity is large, the image may become blurred due to raindrop scattering of light, resulting in a decrease in the performance of the camera. At this time, the first weight coefficient can be reduced, for example, from 0.7 to 0.4, and the second weight coefficient can be increased accordingly, from 0.3 to 0.6, so that the radar data accounts for a larger proportion in the target position estimation, because the radar is less affected by the rainfall.

[0052] In a night environment: when the light sensor detects that the ambient light intensity is very low, the imaging quality of the camera becomes poor, and the first weight coefficient can be reduced from 0.6 to 0.3; while the radar can still normally detect the target at night, and the second weight coefficient can be increased from 0.4 to 0.7.

[0053] On a highway: through the vehicle navigation system, it is acquired that the vehicle is in a high-speed driving state and in a highway scene, at this time the vehicle distance is relatively large, and the target recognition can rely on the long-distance visual information of the camera, and the first weight coefficient can be set to 0.6; although the radar can measure speed and distance, it is affected by the multipath effect, and the second weight coefficient is set to 0.4.

[0054] In an urban road environment: the vehicle is driving on an urban road, and there are many interference factors such as surrounding buildings and pedestrians. For visual targets, although small targets such as pedestrians can be identified, they are easily blocked, and the first weight coefficient can be set to 0.5; the radar can effectively detect vehicle targets, and is less affected by the urban environment, and the second weight coefficient is set to 0.5.

[0055] It should be noted that the environmental information is changing in real time, and the updating of the weight coefficients is also dynamic. The environmental perception system of the vehicle continuously monitors the surrounding environment, and feeds back to the weight adjustment module in real time according to the preset rules, so that the first weight coefficient and the second weight coefficient can adapt to the environmental changes in time, and ensure the accuracy and reliability of the target tracking. For example, when the vehicle drives out of the tunnel (low light, weak visual environment, at this time the first weight coefficient is low) and enters the normal light road, the light sensor senses that the light intensity increases significantly, and the weight adjustment module quickly increases the first weight coefficient, making full use of the high-precision advantage of the camera under good light

[0056] Further, in some embodiments of the present application, before tracking the target object using the coordinates of the visual target and the radar target in the world coordinate system, the method further comprises: obtaining speed information and RCS reflection intensity information of the radar target, and obtaining texture information and color information of the visual target; and comprehensively evaluating the speed information, the RCS reflection intensity information, the texture information and the color information according to a preset speed range, a RCS reflection intensity threshold, preset texture information and a preset color range, to filter out false targets.

[0057] Specifically, in this embodiment, the speed information and the RCS reflection intensity information of the radar target are obtained, wherein the speed information reflects the relative motion speed between the target object and the radar, and the RCS reflection intensity information represents the reflection ability of the target object to the radar wave, which is related to factors such as the shape and material of the target object. For example, a vehicle target usually has a large RCS value, while a small object or a target made of low-reflective material has a relatively small RCS value. And the texture information and the color information of the visual target are obtained, wherein the texture information reflects the texture characteristics of the surface of the target object, such as the body texture of a vehicle, the clothing texture of a pedestrian, etc.; and the color information represents the appearance color of the target object, such as a red vehicle, a blue pedestrian clothing, etc.

[0058] The obtained speed information, RCS reflection intensity information, texture information and color information are comprehensively evaluated. If the speed information of the target object is out of the preset speed range, or the RCS reflection intensity information is lower than the threshold, or the texture information does not match the preset texture information, or the obtained color information does not belong to the preset color range, it is considered that the target may be a false target, which is filtered out. For example, when a plastic bag is blown up in the air by the wind, it may be mistakenly detected by the radar as a target object. The RCS reflection intensity of the plastic bag is usually low, about 0.001 square meters, which is much lower than the RCS value of a vehicle target. Its speed may be slow and unstable in direction, which does not conform to the speed mode of normal vehicle driving. At the same time, the color of the plastic bag may not conform to the preset vehicle color range, and the texture does not conform to the texture characteristics of vehicles or pedestrians. Through comprehensive evaluation of these information, the system identifies the plastic bag as a false target and filters it out. Thus, the accuracy of target identification can be improved.

[0059] Further, in some embodiments of the present application, the method further comprises: obtaining radar point cloud data; and performing motion compensation interpolation processing on the radar point cloud data to realize time synchronization of the radar data and the image data.

[0060] Specifically, in this embodiment, the radar obtains the distance, speed, angle, etc. of the target object by emitting electromagnetic waves and receiving reflected waves, and generates radar point cloud data. These data are presented in the form of point clouds, and each point contains coordinate information (usually x, y, z coordinates in the Cartesian coordinate system) of the target object in three-dimensional space. Further, a vehicle kinematic model can be constructed, combined with IMU (Inertial Measurement Unit) and wheel speed meter data, to predict the pose offset of the camera and the radar, and based on this, the radar point cloud is motion compensated, and a double buffer queue is designed to store the radar point cloud data. For example, when the camera frame rate is 30Hz, the virtual timestamp point cloud can be generated by three times spline interpolation for the radar 20Hz data, so as to supplement the radar data in time, so that the time resolution of the radar point cloud data can match the camera image data, to realize time synchronization.

[0061] In summary, according to the target tracking method based on multiple sensors according to the embodiment of the present application, the image data of the current frame of the target object is obtained by the camera, and the radar data of the current frame of the target object is obtained by the radar, and the visual target of the current frame is determined according to the image data, and the radar target of the current frame is determined according to the radar data, and the predicted target of the current frame of the target object is obtained, so as to respectively convert the predicted target, the visual target and the radar target to the world coordinate system, and then the predicted target is matched with the visual target and the radar target in the world coordinate system, if the predicted target is matched with the visual target successfully, or the predicted target is matched with the radar target successfully, then the target object is tracked using the coordinates of the visual target and the radar target in the world coordinate system, so as to reduce the loss or error tracking target, and effectively improve the stability, accuracy and reliability of target tracking.

[0062] Based on the foregoing target tracking method based on multiple sensors according to the embodiment of the present application, the present embodiment further proposes a computer readable storage medium having a target tracking program based on multiple sensors stored thereon, which is executed by a processor to implement the target tracking method based on multiple sensors according to the foregoing embodiment of the present application.

[0063] According to the computer readable storage medium according to the embodiment of the present application, by adopting the target tracking program based on multiple sensors, the loss or error tracking target can be reduced, and the stability, accuracy and reliability of target tracking can be effectively improved.

[0064] Figure 2 is a block schematic diagram of a target tracking device based on multiple sensors according to the embodiment of the present application.

[0065] Specifically, the multiple sensors include a camera and a radar, such as Figure 2As shown, the multi-sensor-based target tracking device 100 comprises a first acquisition module 10, a determination module 20, a second acquisition module 30, a conversion module 40 and a processing module 50.

[0066] The first acquisition module 10 is configured to acquire image data of a current frame of a target object through a camera and acquire radar data of the current frame of the target object through a radar; the determination module 20 is configured to determine a visual target of the current frame according to the image data and determine a radar target of the current frame according to the radar data; the second acquisition module 30 is configured to acquire a predicted target of the current frame of the target object; the conversion module 40 is configured to convert the predicted target, the visual target and the radar target to a world coordinate system respectively; and the processing module 50 is configured to perform target matching on the predicted target with the visual target and the radar target respectively in the world coordinate system, and if the predicted target matches the visual target successfully or the predicted target matches the radar target successfully, track the target object by using coordinates of the visual target and the radar target in the world coordinate system.

[0067] In some embodiments of the present application, the second acquisition module 30 is specifically configured to acquire a tracking target of the target object in a previous frame; and predict a tracking target of the target object in a current frame according to the tracking target of the target object in the previous frame to obtain the predicted target of the current frame.

[0068] In some embodiments of the present application, the second acquisition module 30 is specifically configured to acquire a speed and an acceleration of the tracking target of the previous frame; and predict a tracking target of the target object in a current frame according to the speed and the acceleration by using a kinematic model to obtain the predicted target of the current frame.

[0069] In some embodiments of the present application, the processing module 50 is specifically configured to acquire a first coordinate and a first weight coefficient of the visual target in the world coordinate system, and acquire a second coordinate and a second weight coefficient of the radar target in the world coordinate system; perform weighted calculation and processing on the first coordinate and the second coordinate according to the first weight coefficient and the second weight coefficient to obtain a target object coordinate; and track the target object according to the target object coordinate.

[0070] In some embodiments of the present application, the processing module 50 is further configured to acquire environmental information around the camera and the radar; and update the first weight coefficient and the second weight coefficient according to the environmental information.

[0071] In some embodiments of the present application, the processing module 50 is further configured to acquire speed information and RCS reflection intensity information of the radar target, and acquire texture information and color information of the visual target; and perform comprehensive evaluation on the speed information, the RCS reflection intensity information, the texture information and the color information according to a preset speed range, an RCS reflection intensity threshold, preset texture information and a preset color range to filter false targets.

[0072] In some embodiments of the present application, the processing module 50 is further configured to acquire radar point cloud data; and perform motion compensation interpolation processing on the radar point cloud data to achieve time synchronization of the radar data and the image data.

[0073] It should be noted that other specific embodiments of the target tracking device based on multiple sensors according to the embodiments of the present application can refer to the specific embodiments of the target tracking method based on multiple sensors according to the embodiments of the present application described above. To reduce redundancy, they will not be described here.

[0074] In summary, according to the target tracking device based on multiple sensors according to the embodiments of the present application, the image data of the current frame of the target object is acquired by the first acquisition module through the camera, and the radar data of the current frame of the target object is acquired by the radar, and the visual target of the current frame is determined according to the image data by the determination module, and the radar target of the current frame is determined according to the radar data, and the predicted target of the current frame of the target object is acquired by the second acquisition module, so as to convert the predicted target, the visual target and the radar target to the world coordinate system by the conversion module, and then the predicted target is matched with the visual target and the radar target in the world coordinate system by the processing module. If the predicted target matches the visual target successfully, or the predicted target matches the radar target successfully, the target object is tracked using the coordinates of the visual target and the radar target in the world coordinate system, so as to reduce the situation of losing or tracking the target incorrectly, and effectively improve the stability, accuracy and reliability of target tracking.

[0075] Figure 3 is a block diagram of a vehicle according to an embodiment of the present application.

[0076] As shown in Figure 3 , the vehicle 1000 includes the target tracking device 100 based on multiple sensors according to the embodiments of the present application described above.

[0077] According to the vehicle according to the embodiments of the present application, by adopting the target tracking device based on multiple sensors according to the embodiments of the present application described above, the situation of losing or tracking the target incorrectly can be reduced, and the stability, accuracy and reliability of target tracking can be effectively improved.

[0078] In addition, other configurations and effects of the vehicle according to the embodiments of the present application are known to those skilled in the art, and to reduce redundancy, they will not be described here.

[0079] It is to be appreciated that the above description and the examples that follow are intended to be illustrative only and that changes can be made to the description, either functionally or chronologically, as well as changes being made concerning the order of implementation. The logic and / or steps represented in the flow diagrams and / or described herein can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus) or a propagation medium. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Note that the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via the optical scanner of a device or device or via an intermediary, such as a facility bureau, then compiled, interpreted, or processed in a suitable manner if necessary, and then stored in a computer storage medium.

[0080] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following techniques, which are well known in the art of making integrated circuits, can be used alone or in any combination: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and other implementations known to those with ordinary skill in the art. These various implementations can include

[0081] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The illustrative appearances of the above-mentioned terms in various places in the specification are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0082] In the description of the application, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application.

[0083] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly specified and limited.

[0084] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0085] In the present application, unless otherwise explicitly specified and limited, the first feature "on" or "under" the second feature can be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature "above", "above" and "above" the second feature can be that the first feature is directly above or obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "below" and "below" the second feature can be that the first feature is directly below or obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0086] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as a limitation on the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A target tracking method based on multiple sensors, characterized in that: The multi-sensor includes a camera and a radar, and the method includes: Acquiring image data of a current frame of a target object through the camera, and acquiring radar data of a current frame of the target object through the radar; determining a visual target of the current frame based on the image data, and determining a radar target of the current frame based on the radar data; Obtaining a predicted target of the target object in the current frame; Converting the predicted target, the visual target, and the radar target to a world coordinate system respectively; Performing target matching on the predicted target with the visual target and the radar target respectively in the world coordinate system; If the predicted target matches the visual target successfully, or the predicted target matches the radar target successfully, the target object is tracked using the coordinates of the visual target and the radar target in the world coordinate system.

2. The target tracking method according to claim 1, characterized in that The obtaining of the predicted target of the current frame of the target object includes: Obtaining the tracking target of the target object in the previous frame; The tracking target of the target object in the current frame is predicted according to the tracking target of the previous frame to obtain the predicted target of the current frame.

3. The target tracking method according to claim 2, characterized in that Predicting the tracking target of the target object in the current frame according to the tracking target of the previous frame includes: Obtaining the speed and acceleration of the tracking target in the previous frame; A kinematic model is used to predict a tracking target of the target object in the current frame according to the velocity and the acceleration to obtain a predicted target of the current frame.

4. The target tracking method according to claim 1, wherein: Tracking the target object using the coordinates of the visual target and the radar target in a world coordinate system includes: Acquire a first coordinate and a first weight coefficient of the visual target in a world coordinate system, and acquire a second coordinate and a second weight coefficient of the radar target in a world coordinate system; Performing weighted calculation processing on the first coordinate and the second coordinate according to the first weight coefficient and the second weight coefficient to obtain the coordinates of the target object; The target object is tracked according to the target object coordinates.

5. The target tracking method according to claim 4, characterized in that: The method further comprises: Acquiring environmental information around the camera and the radar; The first weight coefficient and the second weight coefficient are updated according to the environmental information.

6. The target tracking method according to claim 1, characterized in that: Before tracking the target object using the coordinates of the visual target and the radar target in the world coordinate system, the method further includes: Obtain the speed information and RCS reflection intensity information of the radar target, as well as the texture information and color information of the visual target; According to a preset speed range, an RCS reflection intensity threshold, preset texture information, and a preset color range, the speed information, the RCS reflection intensity information, the texture information, and the color information are comprehensively evaluated to filter out false targets.

7. The target tracking method according to claim 1, characterized in that: The method further comprises: Obtain radar point cloud data; Motion compensation interpolation processing is performed on the radar point cloud data to achieve time synchronization between the radar data and the image data.

8. A computer-readable storage medium, characterized in that A multi-sensor based target tracking program is stored thereon, and when the multi-sensor based target tracking program is executed by a processor, the multi-sensor based target tracking method according to any one of claims 1-7 is implemented.

9. A target tracking device based on multiple sensors, characterized in that: The multi-sensor includes a camera and a radar, and the device includes: a first acquisition module, configured to acquire image data of a current frame of a target object through the camera, and acquire radar data of the current frame of the target object through the radar; a determination module, configured to determine a visual target of a current frame based on the image data, and to determine a radar target of the current frame based on the radar data; A second acquisition module is used to obtain the predicted target of the current frame of the target object; a conversion module, for converting the predicted target, the visual target, and the radar target into a world coordinate system respectively; A processing module is configured to perform target matching on the predicted target with the visual target and the radar target in the world coordinate system; if the predicted target is successfully matched with the visual target, or the predicted target is successfully matched with the radar target, the target object is tracked using the coordinates of the visual target and the radar target in the world coordinate system.

10. A vehicle, characterized in that: Including the multi-sensor based target tracking device as described in claim 9.

Images