A target tracking processing method, device, equipment and medium

Abstract

The application provides a target tracking processing method, device, equipment and medium, comprising: collecting point cloud data of a target and image data of the target; comparing the point cloud data and the image data, adjusting parameter data of a laser radar and a camera, so that the point cloud data and the image data are located in the same coordinate system; synchronizing target information output by the laser radar and target information output by the camera to the same timestamp; comparing the point cloud data of the target with real-time target data and comparing the image data of the target with the real-time target data on the two frames before and after the timestamp; when the point cloud data of the target is associated with the real-time target data, outputting target information by the laser radar and updating target information data, and when the image data of the target is associated with the real-time target data, outputting target information by the camera and updating target information data. The application can improve the detection accuracy of the detected object.

Description

Technical Field

[0001] This application relates to the field of sensor fusion tracking technology for autonomous vehicles, specifically to a target tracking processing method, apparatus, device, and medium. Background Technology

[0002] In the field of autonomous driving, multi-object tracking is a key technology for extracting road information from the surrounding environment. LiDAR and cameras, as the main sensors on autonomous vehicles, bear the heavy responsibility of perceiving the surrounding environment. Their output information, as input to multi-object tracking tasks, often possesses different characteristics. For example, the point cloud information output by LiDAR contains position and depth information, exhibiting sparsity and disorder. Due to the sparsity of point cloud information, LiDAR can only operate within a limited sensing range. Cameras, on the other hand, can output rich visual signals, enabling robust detection of partially occluded objects. When the object to be detected is far away and cannot be reliably located using LiDAR, cameras can effectively locate the object with pixel-level precision within the image domain. However, since a single sensor cannot adequately meet the environmental perception requirements of autonomous driving, there are areas for improvement. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the present invention provides a solution to the above-mentioned technical problems.

[0004] This invention proposes a target tracking processing method, comprising:

[0005] Point cloud data of the target is collected using lidar, and image data of the target is collected using a camera;

[0006] By comparing the point cloud data and the image data, the parameters of the lidar and camera are adjusted so that the point cloud data and the image data are located in the same coordinate system.

[0007] A state model is established for the target information output by the lidar, and an observation model is established for the target information output by the camera. The state model and the observation model are synchronized to the same timestamp.

[0008] In the two frames before and after the timestamp, the point cloud data of the target is compared with the real-time target data, and the image data of the target is compared with the real-time target data.

[0009] When the point cloud data of the target is associated with the real-time target data, the target information is output and updated through the LiDAR; when the image data of the target is associated with the real-time target data, the target information is output and updated through the camera.

[0010] If the point cloud data of the target is not associated with the real-time target, or if the image data of the target is associated with the real-time target, then the point cloud data or the image data is still associated with the real-time target for comparison.

[0011] In one embodiment of the present invention, after the steps of associating and comparing the target's point cloud data with the state model and associating and comparing the target's image data with the observation model in the two frames before and after the timestamp, the method includes:

[0012] The point cloud data and / or image data of the tracked target are recorded as target list data;

[0013] A preset association threshold number of frames is set, within which unassociated targets are deleted from the target list data.

[0014] In one embodiment of the present invention, the step of comparing the point cloud data and the image data, and adjusting the parameter data of the lidar and camera so that the point cloud data and the image data are located in the same coordinate system includes:

[0015] After installing the lidar and camera, place two identical square chessboards directly in front of the lidar and camera respectively, and establish the coordinates of the square chessboards.

[0016] On the coordinates of the checkerboard, calculate the rotation and translation matrices between the point cloud data and the image data, so that the point cloud data and the image data are located in the same coordinate system.

[0017] In one embodiment of the present invention, the steps of establishing a state model for the target information output by the lidar, establishing an observation model for the target information output by the camera, and synchronizing the state model and the observation model to the same timestamp are as follows:

[0018] The target information output by the lidar and the target information output by the camera are synchronized in timestamps, satisfying the following:

[0019] ;

[0020] Among them, X t For the predicted target information results, X t-1 For the target information data at the current moment, u t A is the control variable data for the target information at the current moment. t and B t Let ω be the coefficient matrix. t The mean is 0 and the variance is Q. t Gaussian random variable data.

[0021] In one embodiment of the present invention, the steps of associating and comparing the point cloud data of the target with the real-time target in two frames before and after the timestamp, and associating and comparing the image data of the target with the real-time target data:

[0022] The point cloud data of the target detected by the lidar is correlated and matched with the real-time target data to satisfy:

[0023] ,

[0024] ;

[0025] Wherein d(B i B j α(B) represents the radar output projection distance of the target in the direction of travel. i B j The orientation of the bounding box for the target information represents the target's yaw angle. This includes the target's 3D data and the bounding box's dimensions. Let d(B) be the angle of the bounding box along the coordinate axis. i B j () less than the preset threshold θ max When, it indicates that the point cloud data of the target detected by the lidar is associated with the state model. The bounding box algorithm is an algorithm for finding the optimal bounding space of a discrete point set. Its basic idea is to approximate complex geometric objects with a slightly larger, simpler geometric shape (called a bounding box).

[0026] In one embodiment of the present invention, the satisfy:

[0027] ;

[0028] Where x, y, z are the three-dimensional position data of the target, and h, w, l are the height, width and length data of the bounding box, respectively.

[0029] In one embodiment of the present invention, the steps of associating and comparing the point cloud data of the target with the real-time target data in two frames before and after the timestamp, and associating and comparing the image data of the target with the real-time target data are as follows:

[0030] The image data of the target detected by the camera is correlated and matched with the real-time target data to obtain the overlap data between the image data and the real-time target data;

[0031] When the overlap data is less than a preset overlap threshold, the image data is associated with the real-time target data;

[0032] When the overlap data is greater than or equal to a preset overlap threshold, the image data and the real-time target data continue to be associated and matched.

[0033] The present invention also proposes a target tracking processing device, the device comprising:

[0034] The acquisition unit is used to acquire point cloud data of the target using a lidar and image data of the target using a camera.

[0035] An adjustment unit is used to compare the point cloud data and the image data, and adjust the parameter data of the lidar and the camera so that the point cloud data and the image data are located in the same coordinate system.

[0036] The synchronization unit is used to establish a state model for the target information output by the lidar, establish an observation model for the target information output by the camera, and synchronize the state model and the observation model to the same timestamp.

[0037] The first comparison unit is used to compare the point cloud data of the target with the real-time target data in two frames before and after the timestamp, and to compare the image data of the target with the real-time target data.

[0038] The output unit is used to output target information and update target information data through the lidar when the point cloud data of the target is associated with the real-time target data, and to output target information and update target information data through the camera when the image data of the target is associated with the real-time target data.

[0039] The second comparison unit is used to continue to compare and associate the point cloud data or the image data with the real-time target when the point cloud data of the target is not associated with the real-time target, or when the image data of the target is associated with the real-time target.

[0040] The present invention also proposes an electronic device, the electronic device comprising:

[0041] One or more processors;

[0042] A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to implement the target tracking processing method as described in any of the preceding claims.

[0043] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer processor, causes the computer to perform the target tracking processing method described in any of the preceding claims.

[0044] The beneficial effects of the present invention are: the present invention can accurately and reliably locate objects at different distances, thereby improving the detection accuracy of the objects.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0047] Figure 1 The diagram shows an application environment of the target tracking processing method of the present invention.

[0048] Figure 2 The diagram shown is a flowchart illustrating the target tracking processing method of the present invention.

[0049] Figure 3 This invention is shown as Figure 2 A flowchart illustrating a specific implementation of step S230.

[0050] Figure 4 This invention is shown as Figure 2 A flowchart illustrating a specific implementation of step S220.

[0051] Figure 5 This invention is shown as Figure 2 A flowchart illustrating a specific implementation of step S260.

[0052] Figure 6 This is a schematic diagram of the framework for tracking target information in this invention.

[0053] Figure 7 The diagram shows the structure of the lidar and camera for processing target information in this invention.

[0054] Figure 8 The diagram shown is a structural schematic of the target tracking processing device of the present invention.

[0055] Figure 9 The diagram shown is a structural schematic of a computer device according to an embodiment of the present invention. Detailed Implementation

[0056] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0057] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0058] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0059] First, it's important to clarify that lidar (Light Detection and Ranging) is a radar system that uses laser beams to detect the position, velocity, and other characteristics of a target. Its working principle involves emitting a detection signal (laser beam) towards the target, then comparing the received signal (target echo) with the emitted signal. After appropriate processing, information about the target can be obtained, such as its distance, azimuth, altitude, speed, attitude, and even shape. This allows for the detection, tracking, and identification of targets like aircraft and missiles. It consists of a laser transmitter, an optical receiver, a turntable, and an information processing system. The laser converts electrical pulses into light pulses and emits them. The optical receiver then converts the light pulses reflected from the target back into electrical pulses and sends them to a display. A camera is a device that uses optical imaging principles to form an image and records it using film; it's an optical instrument used for photography. In modern society, many devices can record images and possess characteristics of cameras, such as medical imaging equipment and astronomical observation equipment. The light reflected from the subject passes through the camera lens (viewfinder) and the shutter controls the exposure, and then a latent image of the subject is formed on the photosensitive material in the dark box. After processing (i.e., developing and fixing), a permanent image is formed. This technique is called photography, which is divided into general photography and professional videography.

[0060] Figure 1 This is a schematic diagram illustrating the implementation environment of a target tracking processing method according to an exemplary embodiment of this application. Figure 1 As shown, in some embodiments, the current user of the client can send input commands to the server via a communication network. After receiving the input commands from the client, the server can generate code files based on the automotive platform. Figure 1 The server shown can be any terminal device that supports the installation of navigation map software, such as a smartphone, in-vehicle computer, tablet, laptop, or wearable device, but is not limited to these. Figure 1 The server shown is a server, which can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. No restrictions are placed on this. Clients can communicate with the server via wireless networks such as 3G (third-generation mobile information technology), 4G (fourth-generation mobile information technology), and 5G (fifth-generation mobile information technology), which is also not restricted here.

[0061] In some embodiments, the point cloud information output by the LiDAR includes position and depth information, exhibiting sparsity and disorder. Due to the sparsity of the point cloud information, the LiDAR can only operate within a limited sensing range. A camera, on the other hand, can output rich visual signals to obtain robustness in detecting partially occluded objects. When the object to be detected is far away and cannot be reliably located using LiDAR, the camera can effectively locate the object with pixel-level precision in the image domain. However, since a single sensor cannot adequately meet the environmental perception requirements of autonomous driving, there are areas for improvement. To address these issues, embodiments of this application propose a target tracking processing method, a target tracking processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail below.

[0062] Figure 2 This is a schematic diagram illustrating an implementation environment of a target tracking processing method according to an exemplary embodiment of this application. In some embodiments, the method can be applied to... Figure 1 The implementation environment shown is used, and the method is specifically executed by a client in that implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0063] In some embodiments, for example, the client to which the method for generating code files based on an automotive platform disclosed in this embodiment is applicable may have an SDK (Software Development Kit, a set of development tools for building application software for specific software packages, software frameworks, operating systems, etc.) installed, and the method disclosed in this embodiment is specifically implemented as one or more functions provided by the SDK.

[0064] Please see Figure 2 , Figure 2 This is a flowchart illustrating a target tracking processing method as shown in an exemplary embodiment of this application. This method can be applied to... Figure 1 The implementation environment shown is specifically executed by a smart terminal within that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable.

[0065] For example, the smart terminal to which the traffic condition refresh method disclosed in this embodiment is applicable may have a navigation SDK (Software Development Kit, which is a set of development tools for building application software for specific software packages, software frameworks, operating systems, etc.) installed, and the method disclosed in this embodiment is specifically implemented as one or more functions provided by the navigation SDK.

[0066] like Figure 2 As shown, in an exemplary embodiment, the target tracking processing method includes at least steps S210 to S260, which are described in detail below:

[0067] Step S210: Collect point cloud data of the target using a lidar and image data of the target using a camera.

[0068] In some embodiments, after the lidar and camera are installed, their internal parameters can be preset. Point cloud data of the target can be acquired via the lidar, and image data of the target can be acquired via the camera. Target information output from the lidar and camera can be obtained separately using detection algorithms.

[0069] Step S220: Compare the point cloud data and the image data, and adjust the parameter data of the lidar and camera so that the point cloud data and the image data are located in the same coordinate system.

[0070] In some embodiments, after configuring the parameters of the LiDAR and camera, a checkerboard pattern can be set up directly in front of the LiDAR and camera, respectively. For example, a black and white checkerboard pattern can be set up directly in front of the LiDAR and camera, and the checkerboard pattern can be vertically suspended in front of the LiDAR and camera. When installing the black and white checkerboard pattern, care should be taken to ensure that there are no other obstacles at the suspension location to avoid generating additional noisy point cloud data during LiDAR scanning and calibration. Using the black and white checkerboard pattern as the detection target, feature point data corresponding to the point cloud data and image data are extracted separately. Then, the rotation matrix R and translation matrix T between the LiDAR coordinate system and the camera coordinate system can be calculated using the method of minimizing reprojection error. By transforming the rotation matrix R and translation matrix T, the point cloud data and image data can be set in the same coordinate system.

[0071] Step S230: Establish a state model for the target information output by the lidar, establish an observation model for the target information output by the camera, and synchronize the state model and the observation model to the same timestamp.

[0072] In some embodiments, when synchronizing the target information output by the lidar and the camera, the state transition equations for the state model corresponding to the target information output by the lidar and the observation model corresponding to the target information output by the camera can be:

[0073] .

[0074] Among them, X t For the predicted target information results, X t-1 For the target information data at the current moment, u t A is the control variable data for the target information at the current moment. t and B t Let ω be the coefficient matrix. t The mean is 0 and the variance is Q. t Gaussian random variable data.

[0075] Step S240: On the two frames before and after the timestamp, the point cloud data of the target is compared with the real-time target data, and the image data of the target is compared with the real-time target data.

[0076] In some embodiments, the point cloud data of the target detected by the lidar is correlated and matched with real-time target data to satisfy:

[0077] ,

[0078] ;

[0079] Wherein d(B i Bj α(B) represents the radar output projection distance of the target in the direction of travel. i B j The orientation of the bounding box for the target information represents the target's yaw angle. This includes the target's 3D data and the bounding box's dimensions. Let d(B) be the angle of the bounding box along the coordinate axis. i B j () less than the preset threshold θ max When, it indicates that the point cloud data of the target detected by the lidar is associated with the state model. The bounding box algorithm is an algorithm for finding the optimal bounding space of a discrete point set. Its basic idea is to approximate complex geometric objects with a slightly larger, simpler geometric shape (called a bounding box).

[0080] In some embodiments, the satisfy:

[0081] ;

[0082] Where x, y, z are the three-dimensional position data of the target, and h, w, l are the height, width and length data of the bounding box, respectively.

[0083] Step S250: When the point cloud data of the target is associated with the real-time target data, the target information is output and the target information data is updated through the lidar; when the image data of the target is associated with the real-time target data, the target information is output and the target information data is updated through the camera.

[0084] In some embodiments, distant targets can be identified by receiving 3D detection results from a LiDAR and 2D detection results from a camera. That is, distant targets can be identified using information output from the camera, and when the target enters the LiDAR's sensing range, the LiDAR can quickly and accurately locate the target. Because the target information output by both the LiDAR and the camera is fully utilized throughout the tracking process, and the LiDAR and camera output information are not coupled during use, this application can still function normally even if either the LiDAR or the camera is obstructed or the target tracking information is briefly lost. This application can complete the tracking of distant targets within the image area before the target enters the LiDAR's sensing range.

[0085] Step S260: If the point cloud data of the target is not associated with the real-time target, or the image data of the target is associated with the real-time target, then continue to compare and associate the point cloud data or the image data with the real-time target.

[0086] In some embodiments, when the target's point cloud data is not associated with the real-time target, or the target's image data is associated with the real-time target, it indicates that the LiDAR or camera is tracking the target but cannot achieve real-time synchronization with it. Therefore, it is necessary to continue to associate and compare the point cloud data and image data with the real-time target.

[0087] Please see Figure 3 The diagram illustrates the processing steps for point cloud data and image data according to this application. In some embodiments, step S310 can be executed first to track the point cloud data and / or image data of the target and record it as target list data. Next, step S320 can be executed to preset an association threshold frame number. Within this threshold frame number, targets that have not been associated are deleted from the target list data. When the preset association threshold frame number is exceeded, it indicates that the target has been lost from the field of view of the LiDAR or camera, and the LiDAR or camera can no longer track the target. In this case, unassociated targets can be deleted from the target list data.

[0088] Please see Figure 4 The diagram illustrates the steps involved in aligning the point cloud data from the LiDAR and the image data from the camera within the same coordinate system. In some embodiments, step S410 involves installing the LiDAR and camera, then placing two identical checkerboard tiles directly in front of the LiDAR and camera respectively, establishing checkerboard coordinates. Next, step S420 involves calculating the rotation and translation matrices between the point cloud data and the image data on the checkerboard coordinate system, ensuring that the point cloud data and image data are aligned within the same coordinate system.

[0089] Please see Figure 5 The diagram illustrates a process for associating and matching image data detected by a camera with real-time target data. In some embodiments, step S510 is first executed to associatively match the image data of the target detected by the camera with the real-time target data, obtaining overlap data between the image data and the real-time target data. Next, step S520 is executed to preset an overlap threshold and determine whether the overlap data is less than the overlap threshold. When the overlap data is less than the overlap threshold, it indicates that the image data and target data are associated. Then, when the overlap data is greater than or equal to the overlap threshold, the image data and real-time target data continue to be associated and matched.

[0090] In some embodiments, the lidar target corresponding to the target information detected by the lidar Tracking targets with real-time target information Matching is performed. In the first step of the association process, successfully associated LiDAR targets are... With the tracking target Marked as Unassociated lidar targets Marked as Unassociated tracking targets are marked as After the first step of association, regardless of whether it is an already associated lidar target... Or unrelated lidar targets None of them participate in the second step of association.

[0091] In the second step of the association process, the target information detected by the camera is... Target information detected simultaneously by lidar and camera , and the remaining tracking targets The association is performed based on a two-dimensional IoU (Intersection over Union) calculation. This applies to target information detected by the camera. Tracking targets not associated with the previous step To associate, then Two-dimensional IoU association is performed by projecting the image onto the camera's two-dimensional coordinate system. After the second association step, the successfully associated LiDAR and camera output targets and tracked targets are marked as follows: The unconnected sensor output target is Unsuccessfully associated tracking targets are marked as Sensor targets that have not been associated after the above two steps. Used to create new tracking targets.

[0092] As can be seen, in the above scheme, distant targets can be identified by receiving 3D detection results from the LiDAR and 2D detection results from the camera. That is, distant targets can be identified using the information output by the camera, and when the target enters the LiDAR's sensing range, the LiDAR can quickly and accurately locate the target. Because the target information output by both the LiDAR and the camera is fully utilized throughout the tracking process, and the LiDAR and camera output information do not couple during use, this application can still function normally even if either the LiDAR or the camera is obstructed or the target tracking information is briefly lost. Before the target enters the LiDAR's sensing range, this application can complete the tracking of distant targets within the image area.

[0093] Please see Figure 6The diagram shows a schematic of the structure for processing target tracking information using a lidar and camera. In some embodiments, the lidar sensor 601 can collect point cloud data, and the vision sensor 611 can be a camera used to collect image data. The lidar sensor 601 is electrically connected to the lidar target detection module 602, and the vision sensor 611 is electrically connected to the vision target detection module 612. The lidar target detection module 602 and the vision target detection module 612 are electrically connected to the fusion tracking framework 603. Within the fusion tracking framework 603, TCP (Transmission Control Protocol, Ethernet) receives data information from the lidar target detection module 602 and the vision target detection module 612, and realizes target tracking information through the coordinate transformation module 605, the lidar-vision fusion tracking module 606, and the communication module 607.

[0094] Please see Figure 7 The diagram shows a processing flow chart for lidar target information 701 and camera target information 711. In some embodiments, a spatiotemporal synchronization module 702 can perform synchronization processing of lidar target information 701 and camera target information 711. The lidar target information corresponding to the target information detected by the lidar... 703, tracking target with real-time target information 704 is used for matching. In the first step of the association process, successfully associated LiDAR targets will be... 703 and the tracking target 704 is marked as 706, Unassociated lidar target Marked as 707, Unassociated tracking targets are marked as 705. After the first step of association, regardless of whether it is an already associated lidar target... 706, or an unrelated lidar target. 703 does not participate in the second step of association.

[0095] In the second step of the association process, the target information detected by the camera is... 712 and target information detected simultaneously by lidar and camera , and the remaining tracking targets 713 is used for association, and the association standard is based on two-dimensional IoU (Intersection over Union) calculation. This applies to target information detected by the camera. 712 and the tracking target not associated with the previous step If 705 is associated, then 705 is projected onto the camera's two-dimensional coordinate system for two-dimensional IoU association. After the second step of association, the successfully associated LiDAR and camera output targets and tracked targets are marked as follows: 715, the target output of the unconnected sensor is 716, tracking targets that failed to be associated are marked as 714. For sensor targets that have not been associated after the above two steps... 716 is used to create new tracking targets.

[0096] Please see Figure 8 This invention proposes a target tracking processing device. In some embodiments, the target tracking processing device includes an acquisition unit 801, an adjustment unit 802, a synchronization unit 803, a first comparison unit 804, an output unit 805, and a second comparison unit 806. Detailed descriptions of each functional module are as follows.

[0097] The acquisition unit 801 is used to acquire point cloud data of the target using a lidar and image data of the target using a camera. The adjustment unit 802 is used to compare the point cloud data and the image data, and adjust the parameters of the lidar and camera to ensure that the point cloud data and the image data are in the same coordinate system. The synchronization unit 803 is used to establish a state model for the target information output by the lidar, establish an observation model for the target information output by the camera, and synchronize the state model and the observation model to the same timestamp. The first comparison unit 804 is used to correlate and compare the target's point cloud data with the real-time target data in two frames before and after the timestamp, and to correlate and compare the target's image data with the real-time target data. The output unit 805 is used to output target information and update target information data via the lidar when the target's point cloud data is correlated with the real-time target data, and to output target information and update target information data via the camera when the target's image data is correlated with the real-time target data. The second comparison unit 806 is used to continue to compare and associate the point cloud data or the image data with the real-time target when the point cloud data of the target is not associated with the real-time target, or when the image data of the target is associated with the real-time target.

[0098] Specific limitations regarding the target tracking processing device can be found in the limitations of the target tracking processing method described above, and will not be repeated here. Each module in the aforementioned target tracking processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0099] Figure 9 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 9 The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0100] like Figure 9 As shown, the computer system 900 includes a Central Processing Unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 902 or programs loaded from storage portion 908 into Random Access Memory (RAM) 903, such as performing the methods described in the above embodiments. The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0101] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from them can be installed into storage section 908 as needed.

[0102] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs various functions defined in the system of this application.

[0103] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0106] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the target tracking processing method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0107] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the target tracking processing method provided in the various embodiments described above.

[0108] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A target tracking processing method, characterized in that, include: Point cloud data of the target is collected using lidar, and image data of the target is collected using a camera; By comparing the point cloud data and the image data, the parameters of the lidar and camera are adjusted so that the point cloud data and the image data are located in the same coordinate system. A state model is established for the target information output by the lidar, and an observation model is established for the target information output by the camera. The state model and the observation model are synchronized to the same timestamp. In the two frames before and after the timestamp, the point cloud data of the target is compared with the real-time target data, and the image data of the target is compared with the real-time target data. When the point cloud data of the target is associated with the real-time target data, the target information is output and updated through the LiDAR; when the image data of the target is associated with the real-time target data, the target information is output and updated through the camera. If the point cloud data of the target is not associated with the real-time target, the point cloud data and the real-time target will continue to be associated and compared. In the steps of associating the point cloud data of the target with the real-time target data, outputting target information and updating target information data through the LiDAR, and associating the image data of the target with the real-time target data, outputting target information and updating target information data through the camera: The target information detected by the lidar corresponds to the lidar target. Tracking targets with real-time target information To establish a connection, successfully associated lidar targets will be... With the tracking target Marked as And output via lidar; mark unassociated tracking targets as ; Target information detected by the camera Tracking targets that are not associated Real-time target information tracking target Perform association, and mark the successfully associated camera output targets and tracked targets as... And output through the camera; output the target from the unconnected sensor. And the sensor targets that are still not associated Used to create new tracking targets; In the steps of associating and comparing the point cloud data of the target with the real-time target in two frames before and after the timestamp, and associating and comparing the image data of the target with the real-time target data: The point cloud data of the target detected by the lidar is correlated and matched with the real-time target data to satisfy: ， ； Wherein d(B i B j α(B) represents the radar output projection distance of the target in the direction of travel. i B j The orientation of the bounding box for the target information represents the target's yaw angle. This includes the target's 3D data and the bounding box's dimensions. Let d(B) be the angle of the bounding box along the coordinate axis. i B j () less than the preset threshold θ max When, it indicates that the point cloud data of the target detected by the lidar is associated with the state model.

2. The target tracking processing method according to claim 1, characterized in that, After the steps of associating and comparing the target's point cloud data with the state model and the target's image data with the observation model in the two frames before and after the timestamp, the following steps are included: The point cloud data and / or image data of the tracked target are recorded as target list data; A preset association threshold number of frames is set, within which unassociated targets are deleted from the target list data.

3. The target tracking processing method according to claim 1, characterized in that, The step of comparing the point cloud data and the image data, and adjusting the parameters of the LiDAR and camera to make the point cloud data and the image data lie in the same coordinate system includes: After installing the lidar and camera, place two identical square chessboards directly in front of the lidar and camera respectively, and establish the coordinates of the square chessboards. On the coordinates of the checkerboard, calculate the rotation and translation matrices between the point cloud data and the image data, so that the point cloud data and the image data are located in the same coordinate system.

4. The target tracking processing method according to claim 1, characterized in that, In the step of synchronizing the target information output by the lidar and the target information output by the camera to the same timestamp: A state model is established based on the target information output by the lidar, and an observation model is established based on the target information output by the camera. The state model and the observation model are then synchronized to the same timestamp. Among them, the target information output by the lidar and the target information output by the camera are synchronized in timestamp, satisfying the following: ； Among them, X t For the predicted target information results, X t-1 For the target information data at the current moment, u t A is the control variable data for the target information at the current moment. t and B t Let ω be the coefficient matrix. t The mean is 0 and the variance is Q. t Gaussian random variable data.

5. The target tracking processing method according to claim 1, characterized in that, The satisfy: ； Where x, y, z are the three-dimensional position data of the target, and h, w, l are the height, width and length data of the bounding box, respectively.

6. A target tracking processing device, characterized in that, The device includes: The acquisition unit is used to acquire point cloud data of the target using a lidar and image data of the target using a camera. An adjustment unit is used to compare the point cloud data and the image data, and adjust the parameter data of the lidar and the camera so that the point cloud data and the image data are located in the same coordinate system. The synchronization unit is used to establish a state model for the target information output by the lidar, establish an observation model for the target information output by the camera, and synchronize the state model and the observation model to the same timestamp. The first comparison unit is used to compare the point cloud data of the target with the real-time target data in two frames before and after the timestamp, and to compare the image data of the target with the real-time target data. The output unit is used to output target information and update target information data through the lidar when the point cloud data of the target is associated with the real-time target data, and to output target information and update target information data through the camera when the image data of the target is associated with the real-time target data. The second comparison unit is used to continue to compare and associate the point cloud data with the real-time target when the point cloud data of the target is not associated with the real-time target. In the steps of associating the point cloud data of the target with the real-time target data, outputting target information and updating target information data through the LiDAR, and associating the image data of the target with the real-time target data, outputting target information and updating target information data through the camera: The target information detected by the lidar corresponds to the lidar target. Tracking targets with real-time target information To establish a connection, successfully associated lidar targets will be... With the tracking target Marked as And output via lidar; mark unassociated tracking targets as ; Target information detected by the camera Tracking targets that are not associated Real-time target information tracking target Perform association, and mark the successfully associated camera output targets and tracked targets as... And output through the camera; output the target from the unconnected sensor. And the sensor targets that are still not associated Used to create new tracking targets; In the steps of associating and comparing the point cloud data of the target with the real-time target in two frames before and after the timestamp, and associating and comparing the image data of the target with the real-time target data: The point cloud data of the target detected by the lidar is correlated and matched with the real-time target data to satisfy: ， ； Wherein d(B i B j α(B) represents the radar output projection distance of the target in the direction of travel. i B j The orientation of the bounding box for the target information represents the target's yaw angle. This includes the target's 3D data and the bounding box's dimensions. Let d(B) be the angle of the bounding box along the coordinate axis. i B j () less than the preset threshold θ max When, it indicates that the point cloud data of the target detected by the lidar is associated with the state model.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the target tracking processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the target tracking processing method according to any one of claims 1 to 5.

Images