Fusion positioning method and system

Abstract

The invention discloses a fusion positioning method and system, and the method comprises the steps: responding to a target object appearing in a preset collection region of an image collection unit in the fusion positioning system, and obtaining radar data and image data corresponding to the target object collected by the image collection unit; based on the initial space synchronization parameter of the fusion positioning system, performing space conversion on the radar data to obtain a radar fitting trajectory of the target object in an image coordinate system; comparing the radar fitting trajectory with a visual fitting trajectory of the image data under the image coordinate system to obtain a trajectory comparison result; in response to the situation that the track comparison result does not meet a preset condition, correcting the initial space synchronization parameter to obtain a corrected space synchronization parameter of the fusion positioning system; and based on the corrected space synchronization parameter, fusing the radar data and the image data to obtain fusion positioning information of the target object so as to realize fusion positioning of the target object.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation systems technology, and in particular to a fusion positioning method and system. Background Technology

[0002] With the development of autonomous driving, the application requirements of roadside multi-source fusion positioning systems in vehicle-road cooperative services are becoming increasingly stringent. Building the capability of fusion positioning systems is an important research direction for intelligent transportation services.

[0003] In related technologies, multi-source fusion sensing devices fixed on the roadside typically rely on joint calibration and detection with static target objects. This method is highly dependent on static target objects. However, in real-world scenarios, the static target objects used as references can also move, leading to inaccurate positioning results. Summary of the Invention

[0004] This application provides a fusion positioning method and system to improve the positioning accuracy of target objects.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a fusion positioning method applied to a fusion positioning system. The method includes: in response to the appearance of a target object in a preset acquisition area of ​​an image acquisition unit in the fusion positioning system, acquiring radar data and image data corresponding to the target object acquired by the image acquisition unit; based on the initial spatial synchronization parameters of the fusion positioning system, spatially transforming the radar point cloud data to obtain a radar fitting trajectory of the target object in an image coordinate system; comparing the radar fitting trajectory and the visual fitting trajectory of the image data in the image coordinate system to obtain a trajectory comparison result; in response to the trajectory comparison result not meeting a preset condition, correcting the initial spatial synchronization parameters to obtain corrected spatial synchronization parameters of the fusion positioning system, so as to perform fusion positioning of the target object using the corrected spatial synchronization parameters.

[0007] Secondly, embodiments of this application provide a fusion positioning system, which includes at least an image acquisition unit and a processor; wherein:

[0008] The image acquisition unit is used to acquire radar data and image data corresponding to the target object in response to the appearance of a target object in a preset acquisition area; the processor is used to acquire the radar data and image data corresponding to the target object acquired by the image acquisition unit; based on the initial spatial synchronization parameters of the fusion positioning system, the radar point cloud data is spatially transformed to obtain the radar fitting trajectory of the target object in the image coordinate system; the radar fitting trajectory and the visual fitting trajectory of the image data in the image coordinate system are compared to obtain a trajectory comparison result; in response to the trajectory comparison result not meeting the preset conditions, the initial spatial synchronization parameters are corrected to obtain the corrected spatial synchronization parameters of the fusion positioning system, so as to perform fusion positioning of the target object through the corrected spatial synchronization parameters.

[0009] Thirdly, embodiments of this application provide a fusion positioning device, which includes at least a memory for storing executable instructions; and a processor for implementing the above-mentioned fusion positioning method when executing the executable instructions stored in the memory.

[0010] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed by a processor, implement the aforementioned fusion positioning method.

[0011] Fifthly, embodiments of this application provide a computer program product, the computer program product including executable instructions stored in a computer-readable storage medium; when the processor of the fusion positioning device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned fusion positioning method is implemented.

[0012] The fusion positioning method provided in this application spatially transforms the radar point cloud data acquired by the image acquisition unit based on the initial spatial synchronization parameters of the fusion positioning system to obtain the radar fitting trajectory of the target object in the image coordinate system. Based on the comparison result between the radar fitting trajectory and the visual fitting trajectory, if the preset conditions are not met, the initial spatial synchronization parameters are corrected so that the comparison result between the radar fitting trajectory and the visual fitting trajectory meets the preset conditions. Then, the radar data and image data are fused to obtain the fusion positioning information of the target object. Thus, this embodiment of the application locates the target object based on radar data and image data, obtaining more positioning information and improving the accuracy of target object positioning in real time. Simultaneously, this embodiment of the application corrects the spatial synchronization parameters of the radar data transformation, realizing the self-correction capability of the spatial synchronization parameters of the fusion positioning system during operation, improving the effectiveness and accuracy of the positioning information. Attached Figure Description

[0013] Figure 1 This is an optional framework diagram of the fusion positioning system provided in the embodiments of this application;

[0014] Figure 2 This is an optional flowchart illustrating the fusion positioning method provided in an embodiment of this application;

[0015] Figure 3 This is an optional flowchart illustrating the fusion positioning method provided in an embodiment of this application;

[0016] Figure 4 This is an optional flowchart illustrating the fusion positioning method provided in an embodiment of this application;

[0017] Figure 5 This is an optional flowchart illustrating the fusion positioning method provided in an embodiment of this application;

[0018] Figure 6 This is an optional flowchart illustrating the fusion positioning method provided in an embodiment of this application;

[0019] Figure 7 This is a schematic diagram of the composition structure of the fusion positioning device provided in the embodiments of this application. Detailed Implementation

[0020] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0021] It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0022] Unless otherwise specified, any step in the embodiments of this application performed by the electronic device may be executed by the processor of the electronic device. It is also worth noting that the embodiments of this application do not limit the order in which the electronic device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods. It should also be noted that any step in the embodiments of this application can be executed independently by the electronic device; that is, when the electronic device performs any step in the following embodiments, it may not depend on the execution of other steps.

[0023] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0024] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] With the rapid development of the autonomous driving field, the requirements for the application and implementation of artificial intelligence in autonomous driving are becoming increasingly stringent. Among these, the development of multi-source fusion perception system capabilities based on roadside infrastructure is a crucial area of ​​focus for vehicle-road cooperative autonomous driving. Currently, the typical technical solutions for multi-source fusion perception systems fall into two categories:

[0026] Type 1: The roadside edge cooperative sensing subsystem utilizes a particle filter algorithm to fuse the positioning information of surrounding vehicles acquired by the onboard sensing and positioning subsystem and the positioning information of road environment vehicles acquired by the roadside sensing and positioning subsystem after spatial coordinate transformation. It then performs position compensation on the fused target based on a designed cooperative sensing position and event offset compensation method. However, this type of optimization focuses on the overall solution, resulting in insufficient optimization granularity and limited improvement in positioning accuracy.

[0027] Type Two: When the vehicle is driving outdoors, inertial measurement unit (IMU) data and satellite data are used to determine the vehicle's pose when it is outdoors, indoors, and switching between outdoor and indoor environments. This enables precise vehicle positioning in different scenarios and when switching between them, achieving high-precision vehicle positioning across multiple scenarios. However, this approach is highly dependent on the scenario and location for positioning optimization and cannot achieve real-time and arbitrary sensing area positioning error monitoring.

[0028] The existing technology for target detection and localization in port docking scenarios involves a vehicle information acquisition module that collects vehicle driving information and sends it to a location module and a target detection module. The location module enhances lateral position information based on absolute position in a high-precision map or a local map, as well as structured road surface conditions. The target detection module detects targets near the vehicle and generates main reference line path information and obstacle prediction trajectories based on vehicle information, historical trajectory points, and road boundary information from the high-precision map. A multi-source information synchronization module provides multi-source information time synchronization strategies, multi-sensor data fusion strategies, and inter-frame result prediction methods, thus offering an environmental perception and localization solution for common port scenarios. However, this solution still suffers from insufficient improvement in the localization accuracy of detected targets.

[0029] Based on this, embodiments of this application provide a fusion positioning method and system. The method acquires radar data and image data of a target object. After spatially transforming the radar point cloud data acquired by the image acquisition unit based on the initial spatial synchronization parameters of the fusion positioning system, a radar-fitted trajectory of the target object in the image coordinate system is obtained. If the comparison between the radar-fitted trajectory and the visual-fitted trajectory does not meet preset conditions, the initial spatial synchronization parameters are corrected so that the comparison between the radar-fitted trajectory and the visual-fitted trajectory meets the preset conditions. Then, the radar data and image data are fused to obtain fusion positioning information of the target object. Thus, embodiments of this application locate the target object based on radar data and image data, obtaining more positioning information and improving the accuracy of target object positioning in real time. Simultaneously, embodiments of this application correct the spatial synchronization parameters of the radar data conversion, realizing the self-correction capability of the spatial synchronization parameters during operation of the fusion positioning system, thereby improving the effectiveness and accuracy of the positioning information.

[0030] The following describes an exemplary application of the fusion positioning system according to embodiments of this application. The fusion positioning system may include an image acquisition unit and a processor. The processor may be located in the fusion positioning device and may be implemented as a terminal device in a vehicle road measurement positioning scenario or as a cloud server. In one implementation, the fusion positioning system provided in this application embodiment can be implemented as any terminal with data processing capabilities, such as a laptop, tablet, desktop computer, or mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device). In another implementation, the fusion positioning system provided in this application embodiment can also be implemented as a server. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, which is not limited in this application embodiment.

[0031] This application embodiment can be applied to the field of intelligent transportation technology. As a multi-source sensing and positioning device in the field of vehicle-to-everything (V2X) communication, it is used in a vehicle-road cooperative system as a roadside multi-source sensing and positioning device to acquire radar and image information of target vehicles and fuse them to generate fused positioning information of roadside vehicles. In implementation, the fused positioning device can be installed on both sides of the road or at the edge of a port using either a front-mounted or side-mounted installation method. The front-mounted installation method can be based on the crossbar of a roadside traffic pole, while the side-mounted installation method can be based on the vertical pole of a roadside traffic pole. In this application embodiment, due to the complexity of urban road traffic elements and the large amount of data, considering the recognition accuracy of the fused positioning system and the effective fusion of multi-source sensor data, the front-mounted installation method is chosen when the scenario conditions permit. In application scenarios such as highways and urban expressways, if there is no crossbar, the side-mounted method is chosen.

[0032] The following will describe an exemplary application of the fusion positioning system when implemented as a server.

[0033] See Figure 1 , Figure 1 This is an optional framework diagram of the fusion positioning system provided in the embodiments of this application. The embodiments of this application use the application of the fusion positioning method to a fusion positioning system as an example for illustration.

[0034] The fusion positioning system of this application is used to perceive and locate road vehicles at the roadside end. It fuses the obtained roadside fusion positioning information with the vehicle-mounted perception and positioning subsystem, and transmits the fused vehicle positioning information to the road vehicles through the communication subsystem.

[0035] In this embodiment, the fusion positioning system 10 includes at least an image acquisition unit 100 and a processor 300. The image acquisition unit 100 and the processor 300 can be connected via a network 200, which can be wired or wireless. When connected via a wireless network, the wireless network can be a wide area network (WAN), a local area network (LAN), or a combination of both. The processor 300 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0036] In this embodiment, the image acquisition unit 100 includes various types of sensing and detection devices. In this embodiment, the image acquisition unit 100 may include a camera acquisition device 101 and a radar acquisition device 102. The fusion positioning system 10 acquires radar data and image data of road vehicles in the same time period through the radar acquisition device 102 and the camera acquisition device 101, respectively, and sends the radar data and image data to the processor 300 through the network 200. The processor 300 acquires the radar data and image data of the vehicles. After spatially transforming the radar point cloud data acquired by the image acquisition unit based on the initial spatial synchronization parameters of the fusion positioning system, the radar fitting trajectory of the target object in the image coordinate system is obtained. According to the comparison result between the radar fitting trajectory and the visual fitting trajectory, if the preset conditions are not met, the initial spatial synchronization parameters are corrected so that the comparison result between the radar fitting trajectory and the visual fitting trajectory meets the preset conditions. Then, the radar data and image data are fused to obtain the fusion positioning information of the target object, and the fusion positioning information of the target object is sent to the corresponding vehicle, port, or road control center of a certain area.

[0037] It should be noted that the fusion positioning system of this application embodiment can achieve the positioning accuracy of the fusion positioning system based on the existing roadside equipment and fusion positioning system scheme of related technologies without adding any additional hardware.

[0038] Reference Figure 2 , Figure 2 This is an optional flowchart illustrating the fusion positioning method provided in the embodiments of this application; the following will be combined with... Figure 2The steps shown will be explained. It should be noted that... Figure 2 The fusion positioning method in this paper is illustrated by taking the processor as the execution subject as an example.

[0039] Step S201: In response to the appearance of a target object in the preset acquisition area of ​​the image acquisition unit in the fusion positioning system, acquire the radar data and image data corresponding to the target object acquired by the image acquisition unit.

[0040] In this embodiment of the application, the fusion positioning system can be installed on roadside equipment (i.e., fusion positioning equipment) at the edge of the road or at a traffic hub to obtain real-time fusion positioning information of traffic participants on the road, and to monitor and correct the spatial synchronization parameters of the fusion positioning system in real time, so as to achieve high-precision positioning and stable continuous tracking of traffic participants on the road.

[0041] In this embodiment, when roadside equipment is installed at the edge of a road, considering the positioning accuracy and stable continuous tracking of the target object, the distance between any two fusion positioning devices can be set to no more than a preset distance. Correspondingly, each fusion positioning device can be responsible for managing the acquisition and communication of radar data and image data of all traffic participants within its preset collection area. The vehicle-to-everything (V2X) roadside unit within the fusion positioning device broadcasts the fusion positioning information to vehicles within the preset collection area managed by that roadside device, so that vehicles can adjust their driving status based on the fusion positioning information. The preset collection area managed by each roadside device can refer to an area centered on the roadside device itself and within a preset distance.

[0042] In this embodiment, the fusion positioning system may include an image acquisition unit and a processor. The image acquisition unit may include a radar acquisition device and a camera acquisition device, and the radar acquisition device may be at least one of lidar and millimeter-wave radar.

[0043] In this embodiment, the image acquisition unit can acquire radar data of target vehicles within a preset acquisition area using a radar acquisition device. The radar data can be three-dimensional point cloud data, including at least information such as the vehicle's size, shape, and dimensions, or the target object's location information, speed, driving direction, and relative position to other vehicles. The camera acquisition device can acquire image data of target vehicles within the same preset acquisition area. The image data includes at least semantic data of the target object, such as vehicle attribute information like the target object's type, color, speed, driving direction, and license plate number, as well as the target object's behavior information on the road, such as illegal driving and traffic accidents, and the target object's location information.

[0044] Here, compared with cameras, radar equipment has stronger penetration and anti-interference capabilities. It can effectively collect data on target objects in poor weather or at night when there is insufficient light, enabling simultaneous tracking of multiple target objects and improving positioning accuracy.

[0045] In this embodiment of the application, the target object can be any traffic participant within a preset data collection area, and the traffic participant can be a vehicle on the road. In this embodiment of the application, the traffic participant can also be any participant on the road other than vehicles, such as non-motorized vehicles, pedestrians, etc.

[0046] In this embodiment of the application, when the radar acquisition device acquires radar data containing the target object and the camera acquisition device acquires image data containing the target object, the measurement synchronization of the radar acquisition device and the camera acquisition device is required.

[0047] Here, radar acquisition equipment and camera acquisition equipment can acquire data at the same or different acquisition frequencies, and then the two sets of data are synchronized using the timestamp of the data acquired by one device.

[0048] Step S202: Based on the initial spatial synchronization parameters of the fusion positioning system, perform spatial transformation on the radar point cloud data to obtain the radar fitting trajectory of the target object in the image coordinate system.

[0049] In this embodiment, the radar data collected by the radar device is located in the radar coordinate system, and the image data collected by the camera acquisition device is located in the image coordinate system. The initial spatial synchronization parameter is an inherent parameter of the fusion positioning system, which is used to convert the radar data collected by the radar device to the image coordinate system and then fuse it with the image data as the fusion positioning of the target object.

[0050] To ensure positioning accuracy, the initial spatial synchronization parameters need to be corrected periodically. This correction can be set to a preset cycle, such as every 10 minutes. The data from those 10 minutes is used to correct the initial spatial synchronization parameters. In other words, the fusion positioning system can stop parameter correction after completing the previous cycle; after 10 minutes, it automatically starts the current cycle of positioning parameter correction, repeating this process. This avoids unnecessary resource consumption on the processor caused by continuous correction, and the preset cycle correction method meets the needs of practical applications. It should be noted that before starting the second correction cycle, the relevant calculation data from the previous correction cycle is automatically deleted, and only the spatial synchronization parameters obtained from the last correction cycle are retained as the spatial synchronization parameters for the fusion positioning system.

[0051] In this embodiment of the application, the initial spatial synchronization parameters are initialized and configured in the fusion positioning system. This is used to perform spatial transformation on the radar data containing the target object collected by the radar acquisition device at the beginning stage of roadside perception. Thus, when the pose of the target object may change during program operation, the fusion positioning of this scheme is performed.

[0052] In this embodiment, based on the extrinsic parameters in the initial spatial synchronization parameters, the frame data of the radar data is mapped to the image coordinate system through coordinate transformation to obtain the mapping points. The radar detection box is determined based on the mapping points. Multiple trajectory points of the target object mapped to the image coordinate system are realized by the radar monitoring boxes corresponding to multiple frames of radar data, and then the radar fitting trajectory of the target object in the image coordinate system is obtained by fitting.

[0053] Here, the image coordinate system can be a two-dimensional coordinate system. Radar data is three-dimensional point cloud data, so the three-dimensional point cloud data in the radar coordinate system is first transformed into the two-dimensional image coordinate system.

[0054] Among them, the number of frames of radar data of the target object must meet the preset minimum frame requirement (e.g., at least more than 500 frames) in order to achieve the effectiveness of the fitted trajectory.

[0055] Step S203: Compare the radar fitted trajectory and the visual fitted trajectory of the image data in the image coordinate system to obtain the trajectory comparison result.

[0056] In this embodiment, the initial spatial synchronization parameters include at least intrinsic parameters. These intrinsic parameters are used to project the image data acquired by the camera onto the image coordinate system. The intrinsic parameters include an intrinsic parameter matrix and an extrinsic parameter matrix. The visual fitting trajectory of the image data in the image coordinate system can be achieved through steps S2031 to S2034.

[0057] Step S2031: Based on the intrinsic parameter matrix and the extrinsic parameter matrix, perform an affine transformation on the multi-frame visual synchronization image to obtain the multi-frame visual transformation image of the multi-frame visual synchronization image in the image coordinate system at each radar timestamp.

[0058] In some embodiments, the image data includes at least multiple frames of visual images of the target object in a camera coordinate system, which is a three-dimensional coordinate system based on a camera device.

[0059] Here, the extrinsic parameter matrix is ​​used to transform from the world coordinate system to the camera coordinate system, and the intrinsic parameter matrix is ​​used to transform the 3D camera coordinate system to the 2D image coordinate system. The extrinsic parameter matrix is ​​shown in Equation (1), and the intrinsic parameter matrix is ​​shown in Equation (2):

[0060]

[0061]

[0062] Based on the intrinsic and extrinsic parameters, an affine transformation is performed on multiple frames of visually synchronized images to obtain multiple frames of visually transformed images in the image coordinate system, which can be achieved by formula (3):

[0063]

[0064] Where OXYZ is the world coordinate system, uv is the image coordinate system, Oc-Xc-Yc-Zc is the camera coordinate system, oxy is the physical imaging coordinate system, f is the camera focal length, which is equal to the distance from Oc to O, fx = f / dx, fy = f / dy are the normalized focal lengths on the x-axis and y-axis, respectively, 1 / dx is the number of pixels per unit length in the x-direction, and 1 / dx is the number of pixels per unit length in the y-direction.

[0065] Here, the multiple frames of visually converted images and multiple frames of radar point cloud images have been time-synchronized, meaning that under one radar timestamp, there is one frame of visually converted image and one frame of radar point cloud image.

[0066] Step S2032: Based on the multi-frame visual transformation images, determine multiple visual detection boxes of the target object on the image coordinate system.

[0067] In some embodiments, after obtaining multiple frames of visually transformed images, the target object on each frame of visually transformed images is identified to obtain multiple visual detection boxes of the target object in the image coordinate system.

[0068] Step S2033: Based on the time sequence of the radar timestamps and the multiple visual detection boxes, obtain the visual tracking information of the target object in the image coordinate system.

[0069] In some embodiments, the center point of each visual detection box is used as the position of the target object in the image coordinate system under the current radar timestamp, and the position of the target object corresponding to each radar timestamp is obtained. Based on the time order of the radar timestamps, multiple positions are sorted to finally obtain the visual tracking information of the target object, that is, the position change under different timestamps.

[0070] Step S2034: Perform trajectory fitting on the visual tracking information to obtain the visual fitting trajectory of the target object on the image coordinate system.

[0071] Here, the visual tracking information can be fitted using the least squares method, and the trajectory fitting is shown in Equation (4):

[0072]

[0073] Where n refers to the highest power of the fitting formula, and n is less than the number of trajectory points. A trajectory point is the center point (center_x, center_y) of the visual detection box, where x is the X-axis value of the trajectory point, y is the Y-axis value of the trajectory point, and a i It refers to x i The constant coefficients are b, which is the only constant value in the fitting formula.

[0074] Here, there can be multiple target objects in the image, but each target object has a corresponding identifier (ID). The visual tracking information (i.e., trajectory points) used in a visual fitting trajectory belongs to the same target object ID.

[0075] In this embodiment, comparing the radar-fitted trajectory with the visually fitted trajectory of the image data in the image coordinate system means determining whether there is an error in the initial spatial synchronization parameters by evaluating the difference between the radar-fitted trajectory and the visually fitted trajectory. If the difference between the radar-fitted trajectory and the visually fitted trajectory does not meet a preset condition, i.e., it is greater than an error threshold, it indicates that the initial spatial synchronization parameters have a large error and need to be corrected. If the difference between the radar-fitted trajectory and the visually fitted trajectory meets a preset condition, i.e., it is less than an error threshold, it indicates that the error of the initial spatial synchronization parameters is within the error range and does not need to be corrected. Here, the error threshold can be set manually.

[0076] In some implementations, when comparing radar-fitted trajectories and visual-fitted trajectories, the difference can be obtained by comparing the average distance difference between the radar-fitted trajectory and the visual-fitted trajectory at multiple identical x-coordinates in the image coordinate system.

[0077] Step S204: In response to the trajectory comparison result not meeting the preset conditions, the initial spatial synchronization parameters are corrected to obtain the corrected spatial synchronization parameters of the fusion positioning system.

[0078] In this embodiment, the comparison result of the target object can be categorized into two cases: the comparison result meets the preset conditions, and the comparison result does not meet the preset conditions. The comparison result can be understood as the trajectory error between the radar-fitted trajectory and the visual-fitted trajectory. In this embodiment, the preset conditions can be an error threshold set in the initial configuration of the fusion positioning system. If the trajectory error is greater than the error threshold, it is considered that the initial spatial synchronization parameters have changed significantly and need to be adjusted. The initial spatial synchronization parameters are then corrected to obtain the corrected spatial synchronization parameters of the fusion positioning system. The corrected spatial synchronization parameters ensure that the error between the converted radar-fitted trajectory and the visual-fitted trajectory is less than the error threshold. If the trajectory error is less than or equal to the preset threshold, the current trajectory error is considered to be within an acceptable range. The fusion positioning system can then fuse the target object information carried in the radar data and the target object information carried in the image data to achieve accurate positioning of the target object. It should be noted that the preset threshold can be calibrated according to different application scenarios and system requirements.

[0079] In this embodiment, the mechanism by which the fusion positioning system corrects the initial spatial synchronization parameters can be achieved by adjusting the position of either the radar-fitted trajectory or the visual-fitted trajectory to ensure the comparison results meet preset conditions. The spatial synchronization parameters of the fusion positioning system are then calculated based on the adjusted radar-fitted trajectory. In one implementation, when both the radar-fitted trajectory and the visual-fitted trajectory have errors, either trajectory can be used as a reference for position adjustment, and the other trajectory can be adjusted accordingly. In another implementation, when either the radar-fitted trajectory or the visual-fitted trajectory has errors, another relatively accurate trajectory can be used as a reference for position adjustment, and the trajectory with errors can be adjusted accordingly.

[0080] In some embodiments, the initial spatial synchronization parameters may include a rotation matrix and a translation matrix. Correcting the initial spatial synchronization parameters may refer to adjusting the rotation matrix and the translation matrix so that the error between the converted radar fitted trajectory and the visual fitted trajectory is less than an error threshold.

[0081] Here, when the trajectory comparison results meet the preset conditions, the radar data and image data are fused to obtain the fused positioning information of the target object.

[0082] Step S205: Based on the corrected spatial synchronization parameters, the radar data and image data are fused to obtain the fused positioning information of the target object, so as to achieve the fused positioning of the target object.

[0083] In this embodiment, after obtaining the corrected spatial synchronization parameters of the fusion positioning system, the spatial synchronization parameters can be rechecked after a preset time interval to determine if any errors have occurred. At this time, the previously corrected spatial synchronization parameters are used as the initial spatial synchronization parameters of the fusion positioning system, and the system is again checked for errors. The preset time interval can be 10 minutes. This automatic self-correction of the spatial synchronization parameters improves the positioning accuracy of the target object.

[0084] In other words, in this embodiment, the processor of the fusion positioning system performs spatial transformation on the radar point cloud data of the next target object based on the obtained spatial synchronization parameters, thereby obtaining the radar fitting trajectory of the next target object in the image coordinate system; the radar fitting trajectory and the visual fitting trajectory of the next target object are compared to obtain the comparison result of the next target object; in response to the comparison result not meeting the preset conditions, the spatial synchronization parameters are corrected to obtain new spatial synchronization parameters of the fusion positioning system for fusion positioning of the next target object; during this period, when the obtained comparison result meets the preset conditions, no update is made, that is, the current spatial synchronization parameters are retained for the positioning detection of the next target object.

[0085] Here, radar data includes the size information of the target object, including at least the size, shape, and dimensions of the vehicle, as well as the target object's location information, speed, direction of travel, and relative position with other vehicles. Image data also includes the target object's semantic information, such as vehicle attribute information like type, color, speed, direction of travel, and license plate number, as well as the target object's behavior on the road, such as traffic violations and accidents. It can also be the target object's location information. Fusion refers to fusing the size information included in the radar data and the semantic information included in the image data, provided that the error between the radar fitted trajectory after conversion based on corrected spatial synchronization parameters and the visual fitted trajectory is less than the error threshold, to obtain the target object's fused location information. This process improves the accuracy and comprehensiveness of the location information.

[0086] This application embodiment acquires radar data and image data of the target object. After spatially transforming the radar point cloud data acquired by the image acquisition unit based on the initial spatial synchronization parameters of the fusion positioning system, a radar fitting trajectory of the target object in the image coordinate system is obtained. According to the comparison result between the radar fitting trajectory and the visual fitting trajectory, if the preset conditions are not met, the initial spatial synchronization parameters are corrected so that the comparison result between the radar fitting trajectory and the visual fitting trajectory meets the preset conditions. Then, the radar data and image data are fused to obtain the fused positioning information of the target object. Thus, this application embodiment locates the target object based on radar data and image data, obtains more positioning information, and can improve the positioning accuracy of the target object in real time. At the same time, this application embodiment corrects the spatial synchronization parameters of the radar data conversion, realizes the self-correction capability of the spatial synchronization parameters of the fusion positioning system during operation, and improves the effectiveness and accuracy of the positioning information.

[0087] In some embodiments, to ensure positioning accuracy, it is necessary to synchronize radar data and image data in time. The fusion positioning method provided in this application includes radar data comprising at least multiple frames of radar point cloud data and a radar timestamp corresponding to each frame of radar point cloud data; image data comprising at least multiple frames of visual images and a visual timestamp corresponding to each visual image; and the fusion positioning method may further include step S1:

[0088] Step S1: Using the radar timestamp corresponding to each frame of radar point cloud data as a reference, synchronize the multiple frames of visual images to obtain the multiple frames of visual synchronized images corresponding to each radar timestamp.

[0089] In this embodiment of the application, before spatially converting multiple frames of radar point cloud data based on the initial spatial synchronization parameters of the fusion positioning system, the radar point cloud data acquired by the radar acquisition device and the visual images acquired by the camera acquisition device are synchronized in time series based on the radar timestamp corresponding to each frame of radar point cloud data and the visual timestamp corresponding to each frame of visual image, so as to obtain multiple frames of radar point cloud data and multiple frames of visual images with radar timestamps.

[0090] In this embodiment, the radar acquisition device and the camera acquisition device scan and record their respective radar timestamps and visual timestamps according to their respective frequencies. The fusion positioning system searches for single-frame visual images with relatively close visual timestamps based on the radar timestamps, thereby forming multi-frame radar point cloud data and multi-frame visual images with radar timestamps.

[0091] Correspondingly, step S203 can also be achieved through step S2035:

[0092] Step S2035: Compare the radar fitted trajectory and the visual fitted trajectory of the multi-frame visual synchronization image in the image coordinate system to obtain the trajectory comparison result.

[0093] Figure 3 This is another optional flowchart illustrating the fusion positioning method provided in the embodiments of this application, such as... Figure 3 As shown, step S202 of the fusion positioning method provided in this application embodiment can be implemented through steps S301 to S304:

[0094] Step S301: Based on the time sequence of the radar timestamps, starting from the radar point cloud data corresponding to the first radar timestamp, and based on the initial spatial synchronization parameters, project multiple three-dimensional data points corresponding to the target object in the radar point cloud data onto the image coordinate system to obtain multiple two-dimensional projection points.

[0095] In this embodiment, the image coordinate system is a two-dimensional coordinate system, and the plane corresponding to this coordinate system refers to the plane where the image data acquired by the camera detection device is located.

[0096] In this embodiment, according to the time order of the radar timestamps and based on the initial spatial synchronization parameters, multiple three-dimensional point cloud data corresponding to the target object in each frame of radar point cloud data corresponding to each radar timestamp are sequentially projected onto the image coordinate system where the image data is located, with multiple two-dimensional projection points corresponding to each frame of radar point cloud data in the image coordinate system.

[0097] In some embodiments, the initial spatial synchronization parameters include at least extrinsic parameters, which include a rotation matrix and a translation matrix corresponding to a preset rotation center point. Step S301 can be implemented through steps S3011 to S3013:

[0098] Step S3011: Based on the rotation matrix, rotate the radar point cloud data around the rotation center point to obtain the rotated radar point cloud data.

[0099] Step S3012: Based on the translation matrix, perform translation processing on the rotated radar point cloud data to obtain the transformed radar point cloud data.

[0100] In this embodiment, the rotation matrix rotates the coordinates of the radar point cloud data in the radar coordinate system, and the translation matrix translates the rotated coordinates. The rotation and translation processes can convert the coordinates in the radar coordinate system into the coordinates in the camera coordinate system.

[0101] Here, the rotation matrix can be, but is not limited to, at least one of rotation angle, rotation radians, etc. The rotation center point can be a fixed point preset by those skilled in the art according to the actual situation, and is not limited here. For example, the rotation center point can be the origin in the camera coordinate system.

[0102] Based on this rotation matrix, the radar point cloud data can be rotated around the rotation center point to obtain rotated radar point cloud data. Since the coordinate system's position differs after rotation around different rotation center points, different rotation center points typically correspond to different translation amounts. It should be noted that the translation amount can be zero or a non-zero value; this is not limited. Based on this translation amount, the rotated radar point cloud data can be translated to obtain translated radar point cloud data.

[0103] In implementation, those skilled in the art can use any suitable geometric algorithm to determine the rotation matrix between the radar coordinate system and the camera coordinate system, as well as the translation amount corresponding to the preset rotation center point, according to the actual situation. This disclosure does not limit this.

[0104] Step S3013: Project the converted radar point cloud data onto the image coordinate system to obtain multiple two-dimensional projection points corresponding to the target object.

[0105] After converting the radar point cloud data in the radar coordinate system to the camera coordinate system, the transformation from three-dimensional coordinates to three-dimensional coordinates is completed. Then, the radar point cloud data in the camera coordinate system is projected onto the image coordinate system to achieve the transformation from three-dimensional to two-dimensional, resulting in multiple two-dimensional projection points corresponding to the target object.

[0106] Step S302: Determine the bounding rectangles corresponding to the plurality of two-dimensional projection points as the radar detection box of the target object at the first radar timestamp, until the radar detection box of the target object at each radar timestamp is obtained.

[0107] In this embodiment, the bounding rectangle corresponding to multiple two-dimensional projection points can be the smallest bounding box of the location of multiple two-dimensional projection points. The smallest bounding box corresponding to multiple two-dimensional projection points after the first frame of radar point cloud data is also the radar detection box corresponding to the first radar timestamp. After the radar detection box corresponding to the first radar timestamp is completed, the radar detection box corresponding to the second radar timestamp is determined according to the time sequence of the radar timestamp, until the radar detection box corresponding to each radar timestamp of the target object is obtained.

[0108] In this embodiment, the radar detection box (Lidar_box) can be defined as [center_x1, center_y1, h1, w1, t1]. Wherein, Lidar_box refers to the radar detection box; center_x1 is the X-axis coordinate of the center point of the radar detection box in the image coordinate system; center_y1 is the Y-axis coordinate of the center point of the radar detection box in the image coordinate system; h1 is the height of the radar detection box; w1 is the width of the radar detection box; and t1 is the radar timestamp.

[0109] Step S303: Based on the time sequence of the radar timestamps and the radar detection box corresponding to each radar timestamp of the target object, obtain multiple radar tracking information of the target object in the image coordinate system.

[0110] In this embodiment, the center point of each radar detection box is determined based on the radar detection box corresponding to each radar timestamp of the target object in the image coordinate system. According to the chronological order of the radar timestamps, the tracking trajectory information of the target object in the image coordinate system can be obtained. This tracking trajectory information can be understood as the set of center points of radar detection boxes distributed in the image coordinate system.

[0111] In this embodiment, the tracking trajectory information of the same target object, i.e., a target object with a single ID, can be recorded as a separate unit in the target object trajectory stack (which may be the memory corresponding to the positioning fusion system). The recorded tracking trajectory information of the target object must satisfy at least a preset number of frames. In some embodiments, the preset number may be 15.

[0112] In this embodiment of the application, the fusion positioning system can record the tracking trajectory information of other traffic participants on the road at the same time as a single traffic participant as a separate unit.

[0113] Step S304: Perform trajectory fitting on multiple radar tracking information to obtain the radar fitted trajectory of the target object in the image coordinate system.

[0114] In this embodiment, the least squares method can be used to fit the radar tracking information in the image coordinate system of the target object to obtain the radar fitted trajectory of the target object. During the trajectory fitting process, the trajectory points are the physical center points of the radar detection frame.

[0115] In some implementations, the fitting formula for trajectory fitting of radar tracking information of the target object is shown in formula (5):

[0116]

[0117] Where n refers to the highest power of the fitting formula, and n is less than the number of trajectory points. A trajectory point is the center point (center_x, center_y) of the radar detection box, where x is the X-axis value of the trajectory point, y is the Y-axis value of the trajectory point, and a i It refers to x i The constant coefficients are b, which is the only constant value in the fitting formula.

[0118] The fusion positioning method provided in this application, by projecting radar point cloud data detected by radar acquisition equipment onto the image coordinate system and calculating the minimum bounding box in a specific direction, enables the radar fitting trajectory of the target object to be closer to the real correspondence and actual running trajectory, thereby improving the accuracy of spatial synchronization parameter evaluation and fusion positioning.

[0119] In some embodiments, the radar-fitted trajectory of the target object and the visually fitted trajectory of the image data in the image coordinate system are compared, and the trajectory error between the radar-fitted trajectory and the visually fitted trajectory is calculated to determine the accuracy of the current spatial synchronization parameters. Here, the visually fitted trajectory of the image data in the image coordinate system is obtained by fitting the trajectory of the detection points of the target object in the frame data of the image data. The detection points of the target object can be represented in the form of visual detection boxes, so that the detection points can be obtained from the physical center point of the visual detection boxes.

[0120] In some embodiments, after obtaining the radar fitted trajectory and visual fitted trajectory for each radar timestamp, single-frame trajectory information for that radar timestamp can be formed. The data recording format for each single-frame trajectory information track_info can be track_info = [Lidar_box, Camera_box, track_ID]; thus, the data recording format for the unit trajectory information unit_info_list for each object can be unit_infolist = [track_infot1, track_infot2, ..., track_infot...]. n [; where Lidar_box refers to the radar detection box; Camera_box refers to the visual detection box; track_ID refers to the tracking identification number (ID) of the target object; track_infot1 refers to the trajectory information of the target object at the radar timestamp in the first frame; track_infot2 refers to the trajectory information of the target object at the radar timestamp in the second frame; track_infot n This refers to the trajectory information of the target object at the radar timestamp in the nth frame.

[0121] Figure 4This is another optional flowchart illustrating the fusion positioning method provided in the embodiments of this application, such as... Figure 4 As shown, step S203 of the fusion positioning method provided in this application embodiment can also be implemented by steps S401 to S404:

[0122] Step S401: Based on the radar fitting trajectory and the visual fitting trajectory in the image coordinate system, determine multiple second radar coordinates and multiple second visual coordinates corresponding to the radar fitting trajectory and the visual fitting trajectory at multiple first coordinates.

[0123] In this embodiment of the application, the image coordinate system is a two-dimensional coordinate system. After projecting both radar data and image data onto the image coordinate system, multiple first coordinates are obtained on the image coordinate system. These multiple first coordinates can be x-coordinate points on the X-axis of the image coordinate system. The multiple first coordinates can be evenly or unevenly distributed on the X-axis of the image coordinate system, and the radar fitting trajectory and the visual fitting trajectory have corresponding trajectory points on the multiple first coordinates.

[0124] Here, the first coordinate is explained using the X-coordinate as an example. The first coordinate can also be the Y-coordinate, depending on the establishment of the image coordinate system and the projection result. Usually, the direction of the trajectory is used as the coordinate axis of the first coordinate.

[0125] In this embodiment of the application, for each first coordinate, the coordinate points on the radar fitting trajectory and the visual fitting trajectory in the Y direction corresponding to the first coordinate are respectively determined as the second radar coordinate and the second visual coordinate, thereby obtaining the second radar coordinate and the second visual coordinate corresponding to multiple first coordinates.

[0126] Step S402: Compare the second radar coordinate and the second visual coordinate corresponding to each first coordinate to obtain the second coordinate difference between the radar fitting trajectory and the visual fitting trajectory at each first coordinate.

[0127] In this embodiment of the application, for each first coordinate, the difference between the second visual coordinate and the second radar coordinate in the Y direction corresponding to the first coordinate is determined as the coordinate difference between the radar fitting trajectory and the visual fitting trajectory at the first coordinate, that is, the trajectory deviation between the radar fitting trajectory and the visual fitting trajectory under the same radar timestamp, thereby obtaining the coordinate difference between the radar fitting trajectory and the visual fitting trajectory at each first coordinate.

[0128] Step S403: Based on each second coordinate difference and the number of first coordinates, determine the trajectory error between the radar fitted trajectory and the visual fitted trajectory.

[0129] In this embodiment of the application, the integral sum of the coordinate differences at each first coordinate is used as the ratio of the number of first coordinates to the number of first coordinates to determine the trajectory error between the radar fitted trajectory and the visual fitted trajectory.

[0130] In this embodiment, the trajectory error between the radar-fitted trajectory and the visual-fitted trajectory can be determined by the following formula (6):

[0131]

[0132] Where track_error refers to the trajectory error, y camera This refers to the visual fitting trajectory, y lidar This refers to the radar fitted trajectory, y camera -y lidar It refers to the difference in the Y-axis value corresponding to the same coordinate point on the X-axis between the visual fitting trajectory and the radar fitting trajectory. x refers to the first coordinate, and N refers to the number of values ​​that the first coordinate can take. x can be randomly selected from the overlapping segments on the X-axis covered by the radar fitting trajectory and the visual fitting trajectory.

[0133] Step S404: Determine the trajectory error as the trajectory comparison result.

[0134] In this embodiment of the application, the trajectory error between the radar fitted trajectory and the visual fitted trajectory is determined as the comparison result of the target object.

[0135] The fusion positioning method provided in this application, after acquiring radar data and image data and fitting the trajectory of a moving target object, can accurately quantify the error between the acquired radar-fitted trajectory and the visual-fitted trajectory based on the difference calculation between the obtained radar-fitted trajectory and the visual-fitted trajectory. This allows the method to determine whether there is an error in the spatial synchronization parameters currently used by the fusion positioning system, making the error evaluated by the fusion positioning method of this application more realistically reflect the errors existing in the operation of the fusion positioning system.

[0136] Figure 5 This is another optional flowchart illustrating the fusion positioning method provided in the embodiments of this application, such as... Figure 5 As shown, in the fusion positioning method provided in this application embodiment, step S204 can be implemented through steps S501 to S504:

[0137] Step S501: Adjust the rotation matrix and translation matrix in the initial spatial synchronization parameters multiple times. After each adjustment, obtain the adjusted rotation matrix, the adjusted translation matrix, and the corresponding adjusted radar fitting trajectory.

[0138] In this embodiment, if the trajectory error between the radar-fitted trajectory and the visual-fitted trajectory is greater than or equal to the error threshold, it indicates that the pose of the radar acquisition device or the camera acquisition device may have changed, which may have led to a significant change in the initial spatial transformation parameters. Therefore, automatic evaluation and correction are required.

[0139] In this embodiment, the extrinsic parameters in the initial spatial synchronization parameters include rotation and translation matrices, and position adjustment can include adjustment methods in the form of rotation and translation. In one implementation, each time a position adjustment is performed, the radar fitted trajectory is adjusted as close as possible to the visual fitted trajectory through multiple position adjustments, by referencing the position of the visual fitted trajectory; simultaneously, after each position adjustment, the processor calculates the trajectory error between the adjusted radar fitted trajectory and the visual fitted trajectory, compares this trajectory error with an error threshold, and corrects the rotation and translation matrices based on the rotation and translation that occur when the two trajectories are closest.

[0140] Here, after each adjustment, we can obtain the adjusted rotation matrix, the adjusted translation matrix, and the corresponding adjusted radar fitting trajectory.

[0141] Step S502: Match each adjusted radar fitting trajectory with the visual fitting trajectory to obtain multiple matching degrees.

[0142] In this embodiment, the adjusted radar fitting trajectory is matched with the visual fitting trajectory after each adjustment to obtain multiple matching degrees. The matching is also compared using the trajectory error calculation method described above, which will not be repeated here.

[0143] Step S503: The adjustment rotation matrix and adjustment translation matrix corresponding to the adjustment radar fitting trajectory with the highest matching degree are determined as the correction extrinsic parameters.

[0144] In this embodiment, the adjustment rotation matrix and adjustment translation matrix corresponding to the radar fitting trajectory with the highest matching degree (i.e., the smallest trajectory error) among multiple matching degrees can be determined as the correction extrinsic parameters.

[0145] Step S504: Determine the modified extrinsic parameters and the intrinsic parameters in the initial spatial synchronization parameters as the modified spatial synchronization parameters of the fusion positioning system.

[0146] In this embodiment of the application, the modified extrinsic parameters and the intrinsic parameters in the initial spatial synchronization parameters are determined as the modified spatial synchronization parameters of the fusion positioning system, so that the fusion positioning system can locate the target object using the modified spatial synchronization parameters.

[0147] In this embodiment, the processor of the fusion positioning system performs fusion positioning of radar data and image data of the next target object based on the corrected spatial synchronization parameters.

[0148] In some embodiments, step S204 can also be implemented by steps S2041 to S2044:

[0149] Step S2041: Rotate and translate the radar fitting trajectory until the trajectory error between the adjusted radar fitting trajectory and the visual fitting trajectory is less than a preset error threshold, and determine the adjusted radar fitting trajectory as the target radar fitting trajectory of the target object.

[0150] Here, matching points on the visual fitting trajectory can be randomly selected, and the translation matrix of the radar fitting trajectory projected by the LiDAR can be adjusted according to this set of matching points to make the set of matching points coincide. Then, with the coincident points as the center, the rotation matrix is ​​adjusted to rotate the trajectory projected by the LiDAR into the image. This can be done by automatic clockwise rotation to maximize the overlap between the visual fitting trajectory and the radar fitting trajectory, thus completing the adjustment of the spatial synchronization parameter.

[0151] Here, after multiple matches, when the trajectory error between the adjusted radar fitting trajectory and the visual fitting trajectory is less than the error threshold, the adjustment is stopped, and the adjustment rotation matrix and adjustment translation matrix corresponding to the current adjusted radar fitting trajectory with a trajectory error less than the error threshold are determined as the correction extrinsic parameters.

[0152] Here, 100 matching points can be selected, and the above operation can be repeated to obtain the matching degree. Finally, the radar fitting trajectory with the highest matching degree is selected as the target radar fitting trajectory of the target object.

[0153] Step S2042: Extract the second radar coordinates corresponding to multiple first coordinates from the target radar fitting trajectory.

[0154] Step S2043: Based on multiple second radar coordinates, correct the extrinsic parameters in the initial spatial synchronization parameters to obtain corrected extrinsic parameters.

[0155] The translation and rotation matrices can be corrected based on the coordinates in the target radar fitted trajectory to obtain the corrected extrinsic parameters.

[0156] Step S2044: Determine the modified extrinsic parameters and the intrinsic parameters in the initial spatial synchronization parameters as the modified spatial synchronization parameters of the fusion positioning system.

[0157] The fusion positioning method provided in this application adjusts the radar fitting trajectory in the comparison results that do not meet the preset conditions, calculates the corresponding points of the visual coordinates of the visual fitting trajectory and the radar coordinates on the adjusted radar fitting trajectory, and obtains the correction parameters. This enables the parameter self-correction of the fusion positioning system and improves the accuracy of fusion positioning.

[0158] In some embodiments, step S205 can be implemented by steps S2051 and S2052:

[0159] Step S2051: Based on the corrected spatial synchronization parameters, spatially transform the radar data to obtain the corrected radar fitting trajectory of the target object in the image coordinate system.

[0160] Step S2052: In response to the trajectory comparison result between the corrected radar fitting trajectory and the visual fitting trajectory satisfying the preset conditions, the size information and the semantic information are fused to obtain the fused positioning information.

[0161] Here, radar data includes the size information of the target object, including at least the size, shape, and dimensions of the vehicle, as well as the target object's location information, speed, direction of travel, and relative position with other vehicles. Image data also includes the target object's semantic information, such as vehicle attribute information like type, color, speed, direction of travel, and license plate number, as well as the target object's behavior on the road, such as traffic violations and accidents. It can also be the target object's location information. Fusion refers to fusing the size information included in the radar data and the semantic information included in the image data, provided that the error between the radar fitted trajectory after conversion based on corrected spatial synchronization parameters and the visual fitted trajectory is less than the error threshold, to obtain the target object's fused location information. This process improves the accuracy and comprehensiveness of the location information.

[0162] The following embodiment of this application provides an application of the fusion positioning method in a real-world scenario.

[0163] Based on the foregoing embodiments, this application provides a fusion positioning method applied to a fusion positioning system. The method involves acquiring sensing data through an image acquisition unit, storing and processing the sensing data through a processor to obtain vehicle-road tracking information of traffic participants on the road. This vehicle-road tracking information is then used to control the behavior of traffic participants on the road, thereby strengthening traffic behavior regulation. The fusion positioning system includes an initialization configuration module, a fusion sensing module, a trajectory matching module, a trajectory fitting module, a target object trajectory recording module, and a real-time automatic correction module for spatial synchronization parameters. (Refer to...) Figure 6As shown, the method includes steps S601 to S607:

[0164] Step S601: Initialize the configuration module to configure the initial system parameters.

[0165] In this embodiment, the information configured by the initialization configuration module may include, but is not limited to: initial spatial synchronization parameters and the common sensing area corresponding to the radar acquisition device and the camera acquisition device. In this embodiment, the radar acquisition device may be a lidar.

[0166] Step S602: The fusion sensing module acquires radar data and image data.

[0167] In this embodiment, the fusion perception module acquires radar data and images acquired by the radar acquisition device and the camera acquisition device, and synchronizes the acquired radar data and image data in time series to obtain synchronized multi-frame radar point cloud data and multi-frame visual images.

[0168] Step S603: The fusion perception module identifies the synchronized multi-frame radar point cloud data and multi-frame visual images to obtain multiple point cloud detection boxes in the multi-frame radar point cloud data containing the target object and multiple visual detection boxes in the multi-frame image data containing the target object.

[0169] In this embodiment of the application, target object recognition is performed on the synchronized multi-frame radar point cloud data and multi-frame visual images to obtain multiple point cloud detection boxes of the target object in the multi-frame radar point cloud data and multiple visual detection boxes of the target object in the multi-frame image data. Based on the multiple point cloud detection boxes and multiple visual detection boxes, information such as the location, object category and running speed of the target object is obtained.

[0170] Step S604: The trajectory matching module performs trajectory matching based on multiple point cloud detection boxes in multi-frame radar point cloud data containing the target object and multiple visual detection boxes in multi-frame image data containing the target object, and obtains and records the matching results.

[0171] In this embodiment, the trajectory matching module projects the point cloud detection boxes of common sensing areas in the radar point cloud data onto an image (which can be a visual image synchronized with the radar point cloud data in time) according to the initial spatial synchronization parameters configured during initialization, obtaining multiple two-dimensional projection points. The minimum bounding box of all projection points is then calculated to obtain the projection matching detection box (i.e., the radar detection box). The characteristic requirements of the projection matching detection box projected onto the image coordinate system are described as follows:

[0172] 1) The long and wide sides of the projective matching detection box are parallel to the long and wide sides of the visual detection box, respectively; 2) Definition of the projective matching detection box: Lidar_box = [center_x1, center_y1, h1, w1, t1], where: Lidar_box refers to the projective matching detection box; center_x1: the X-axis coordinate of the center point of the projective matching detection box in the image coordinate system; center_y1: the Y-axis coordinate of the center point of the projective matching detection box in the image coordinate system; h1: the height of the projective matching detection box; w1: the width of the projective matching detection box; t1: the frame timestamp of the LiDAR. 3) The definition of the visual detection box is Camera_box = [center_x2, center_y2, h2, w2, t2], where: Camera_box refers to the visual detection box; center_x2: the X-axis coordinate of the center point of the visual detection box in the image coordinate system; center_y2: the Y-axis coordinate of the center point of the visual detection box in the image coordinate system; h2: the height of the visual detection box; w2: the width of the visual detection box; t2: the frame timestamp of the visual detection box.

[0173] Based on the single-frame radar point cloud data and the projected matching detection boxes and visual detection boxes in the visual images obtained above, the trajectory matching module iteratively performs projected matching on each frame of radar point cloud data and the corresponding visual image to obtain the projected matching detection boxes for each frame. In this embodiment, after completing the trajectory matching task for the target object, the trajectory matching module adds the time series of all projected matching detection boxes for a single target object based on the radar timestamp to the target object trajectory stack_list of the target object trajectory recording module.

[0174] In this embodiment of the application, the definition and structure of the data record created by the target object trajectory recording module are as follows: 1) Single frame trajectory information data format track_info = [Lidar_box, Camera_box, track_ID]; unit information format unit_info_list = [track_infot1, track_infot2, ..., track_infot...] n ], where Lidar_box: target object radar detection box information output by the fusion perception module; Camera_box: target object visual detection box information output by the fusion perception module; track_ID: target object tracking ID output by the fusion perception module; 2) In the single frame trajectory information, Lidar_box and Camera_box cannot be empty lists at the same time, and track_ID must have a value.

[0175] Step S605: The trajectory fitting module performs trajectory fitting based on the matching results to obtain the visually fitted trajectory and the radar fitted trajectory.

[0176] In this embodiment, the trajectory fitting module performs trajectory fitting on all projected matching detection boxes and all visual detection boxes of the target object in the image coordinate system to obtain the visual fitting trajectory and the radar fitting trajectory.

[0177] Here, the trajectory fitting module can start its trajectory fitting function every 10 minutes, and cyclically obtain a unit_info_list from the target object trajectory stack_list of the target object trajectory recording module. If the number of trajectories is less than 15, it will wait until the number of trajectories is greater than 15 before starting the fitting.

[0178] In this embodiment of the application, the fitting formulas for trajectory fitting of all projected matching detection boxes and all visual detection boxes of the target object are shown in formula (7):

[0179]

[0180] Where n refers to the highest power of the fitting formula, and n is less than the number of trajectory points. A trajectory point is the center point (center_x, center_y) of the visual detection box, where x is the X-axis value of the trajectory, y is the Y-axis value of the trajectory, and a i It refers to x i The constant coefficient, b, refers to the unique constant value in the fitting formula.

[0181] Step S606: The real-time automatic correction module for spatial synchronization parameters evaluates the trajectory error of the visually fitted trajectory and the radar fitted trajectory to obtain the trajectory error.

[0182] In this embodiment of the application, based on the visual fitting trajectory and radar fitting trajectory obtained in step S605, the trajectory error is calculated using the following formula (8):

[0183]

[0184] Where track_error refers to the trajectory error, y camera This refers to the visual fitting trajectory, y lidar This refers to the radar fitted trajectory, y camera -y lidar It refers to the difference in the Y-axis value corresponding to the same coordinate point on the X-axis between the visual fitting trajectory and the radar fitting trajectory. x refers to the first coordinate, and N refers to the number of values ​​that the first coordinate can take. x can be randomly selected from the overlapping segments on the X-axis covered by the radar fitting trajectory and the visual fitting trajectory.

[0185] Step S607: The real-time automatic correction module for spatial synchronization parameters corrects the initial spatial synchronization parameters based on the trajectory error to obtain spatial synchronization parameters for detecting the next target object.

[0186] In this embodiment, the trajectory error obtained in step S606 is compared with an error threshold. If the trajectory error is greater than or equal to the error threshold, it is considered that the initial spatial synchronization parameters have changed significantly, and error adjustment is required. If the trajectory error is less than the error threshold, it is considered that the current trajectory error is within an acceptable range. Here, the error threshold is included in the information configured by the initialization configuration module.

[0187] In this embodiment, when the trajectory error is greater than or equal to an error threshold, the radar fitted trajectory is adjusted for error. This adjustment is achieved by repeatedly rotating and translating the radar fitted trajectory. The trajectory error is calculated after each adjustment operation until the trajectory error is less than the error threshold. The adjusted radar fitted trajectory is then identified as the target fitted trajectory. Finally, a real-time automatic correction module for spatial synchronization parameters randomly extracts at least 15 sets of corresponding image coordinates from the target radar fitted trajectory, using the lidar coordinate system, lidar point cloud, and image. This allows for the calculation of the corresponding spatial synchronization parameters, thus achieving parameter correction.

[0188] In this embodiment of the application, the process of calculating the corresponding spatial synchronization parameters can be implemented in the following way:

[0189] Randomly select matching points on the visually fitted trajectory, and adjust the translation parameter T (i.e., the translation parameter) of the radar fitted trajectory projected by the LiDAR according to this set of matching points, so that the set of matching points coincide. Then, rotate the trajectory projected by the LiDAR into the image with the coincident point as the center, which can be an automatic clockwise rotation, to maximize the overlap between the visually fitted trajectory and the radar fitted trajectory, thus completing the adjustment of the spatial synchronization parameter. Here, 100 matching points can be selected, and the above operation can be repeated to obtain the matching degree. Finally, select the parameter with the highest matching degree, which is the corrected spatial synchronization parameter.

[0190] Based on the foregoing embodiments, this application provides a fusion positioning device. Figure 7 This is a schematic diagram of the composition structure of the fusion positioning device provided in the embodiments of this application, as shown below. Figure 7 As shown, the fusion positioning device 700 includes: an acquisition module 701, a spatial conversion module 702, a comparison module 703, a correction module 704, and a fusion module 705.

[0191] The acquisition module 701 is used to acquire radar data and image data corresponding to the target object acquired by the image acquisition unit in response to the appearance of a target object in the preset acquisition area of ​​the image acquisition unit in the fusion positioning system; the spatial transformation module 702 is used to perform spatial transformation on the radar data based on the initial spatial synchronization parameters of the fusion positioning system to obtain the radar fitting trajectory of the target object in the image coordinate system; the comparison module 703 is used to compare the radar fitting trajectory and the visual fitting trajectory of the image data in the image coordinate system to obtain a trajectory comparison result; the correction module 704 is used to correct the initial spatial synchronization parameters in response to the trajectory comparison result not meeting the preset conditions to obtain the corrected spatial synchronization parameters of the fusion positioning system; the fusion module 705 is used to fuse the radar data and image data based on the corrected spatial synchronization parameters to obtain the fusion positioning information of the target object, so as to realize the fusion positioning of the target object.

[0192] In some embodiments, the radar data includes at least multiple frames of radar point cloud data of the target object in the radar coordinate system and a radar timestamp corresponding to each frame of radar point cloud data; the image data includes at least multiple frames of visual images of the target object in the camera coordinate system and a visual timestamp corresponding to each frame of visual images; the device further includes: a synchronization module, used to synchronize the multiple frames of visual images based on the radar timestamp corresponding to each frame of radar point cloud data, to obtain multiple frames of synchronized visual images corresponding to each radar timestamp; correspondingly, a comparison module 703 is used to compare the radar fitted trajectory and the visual fitted trajectory of the multiple frames of synchronized visual images in the image coordinate system to obtain a trajectory comparison result.

[0193] In some embodiments, the spatial transformation module 702 is further configured to, based on the temporal order of the radar timestamps, starting from the radar point cloud data corresponding to the first radar timestamp, and based on the initial spatial synchronization parameters, project multiple three-dimensional data points corresponding to the target object in the radar point cloud data onto the image coordinate system to obtain multiple two-dimensional projection points; determine the bounding rectangles corresponding to the multiple two-dimensional projection points as the radar detection boxes of the target object at the first radar timestamp, until the radar detection boxes of the target object at each radar timestamp are obtained; based on the temporal order of the radar timestamps and the radar detection boxes of the target object at each radar timestamp, obtain multiple radar tracking information of the target object in the image coordinate system; and perform trajectory fitting on the multiple radar tracking information to obtain the radar fitting trajectory of the target object in the image coordinate system.

[0194] In some embodiments, the initial spatial synchronization parameters include at least extrinsic parameters, including a rotation matrix and a translation matrix corresponding to a preset rotation center point; the spatial transformation module 702 is further configured to rotate the radar point cloud data around the rotation center point based on the rotation matrix to obtain the rotated radar point cloud data; translate the rotated radar point cloud data based on the translation matrix to obtain the transformed radar point cloud data; and project the transformed radar point cloud data onto the image coordinate system to obtain multiple two-dimensional projection points corresponding to the target object.

[0195] In some embodiments, the initial spatial synchronization parameters include at least intrinsic parameters, which include an intrinsic parameter matrix and an extrinsic parameter matrix. The apparatus further includes: an affine transformation module, configured to perform an affine transformation on the multi-frame visual synchronization images in the camera coordinate system based on the intrinsic parameter matrix and the extrinsic parameter matrix, to obtain a multi-frame visual transformation image of the multi-frame visual synchronization images in the image coordinate system at each radar timestamp; determine multiple visual detection boxes of the target object in the image coordinate system based on the multi-frame visual transformation image; obtain visual tracking information of the target object in the image coordinate system based on the time sequence of the radar timestamps and the multiple visual detection boxes; and perform trajectory fitting on the visual tracking information to obtain a radar-fitted trajectory of the target object in the image coordinate system.

[0196] In some embodiments, the comparison module 703 is further configured to: determine, based on the radar fitting trajectory and the visual fitting trajectory in the image coordinate system, a plurality of second radar coordinates and a plurality of second visual coordinates corresponding to a plurality of first coordinates, respectively; compare the second radar coordinates and the second visual coordinates corresponding to each first coordinate to obtain a second coordinate difference between the radar fitting trajectory and the visual fitting trajectory at each first coordinate; determine the trajectory error between the radar fitting trajectory and the visual fitting trajectory based on each second coordinate difference and the number of first coordinates; and determine the trajectory error as the trajectory comparison result.

[0197] In some embodiments, the correction module 704 is further configured to rotate and translate the radar fitting trajectory until the trajectory error between the adjusted radar fitting trajectory and the visual fitting trajectory is less than an error threshold, and determine the adjusted radar fitting trajectory as the target radar fitting trajectory of the target object; extract the second radar coordinates corresponding to multiple first coordinates from the target radar fitting trajectory; based on the multiple second radar coordinates, correct the extrinsic parameters in the initial spatial synchronization parameters to obtain corrected extrinsic parameters; and determine the corrected extrinsic parameters and the intrinsic parameters in the initial spatial synchronization parameters as the corrected spatial synchronization parameters of the fusion positioning system.

[0198] In some embodiments, the correction module 704 is further configured to adjust the rotation matrix and translation matrix in the initial spatial synchronization parameters multiple times, and after each adjustment, obtain the adjusted rotation matrix, the adjusted translation matrix and the corresponding adjusted radar fitting trajectory; match each adjusted radar fitting trajectory with the visual fitting trajectory to obtain multiple matching degrees; determine the adjusted rotation matrix and the adjusted translation matrix corresponding to the adjusted radar fitting trajectory with the highest matching degree as the correction extrinsic parameters; and determine the correction extrinsic parameters and the intrinsic parameters in the initial spatial synchronization parameters as the correction spatial synchronization parameters of the fusion positioning system.

[0199] In some embodiments, the radar data further includes the size information of the target object, and the image data further includes the semantic information of the target object; the fusion module 705 is further configured to perform spatial transformation on the radar data based on the corrected spatial synchronization parameters to obtain the corrected radar fitting trajectory of the target object in the image coordinate system; in response to the trajectory comparison result between the corrected radar fitting trajectory and the visual fitting trajectory satisfying a preset condition, the size information and the semantic information are fused to obtain the fused positioning information.

[0200] Based on the foregoing embodiments, this application provides a fusion positioning system, which includes at least an image acquisition unit and a processor; wherein: the image acquisition unit is used to acquire radar data and image data corresponding to the target object in response to the appearance of a target object in a preset acquisition area; the processor is used to acquire the radar data and image data corresponding to the target object acquired by the image acquisition unit; based on the initial spatial synchronization parameters of the fusion positioning system, the radar point cloud data is spatially transformed to obtain the radar fitting trajectory of the target object in the image coordinate system; the radar fitting trajectory and the visual fitting trajectory of the image data in the image coordinate system are compared to obtain a trajectory comparison result; in response to the trajectory comparison result not meeting a preset condition, the initial spatial synchronization parameters are corrected to obtain the corrected spatial synchronization parameters of the fusion positioning system, so as to perform fusion positioning of the target object through the corrected spatial synchronization parameters.

[0201] Based on the foregoing embodiments, this application provides a fusion positioning device, including a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the above-mentioned fusion positioning method.

[0202] Based on the foregoing embodiments, this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors, using the steps in the fusion positioning method provided in the above embodiments.

[0203] Based on the foregoing embodiments, this application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; when the processor of the fusion positioning device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned fusion positioning method is implemented.

[0204] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.

[0205] This application is described with flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowcharts and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0206] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0207] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0208] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A fusion positioning method, characterized by, The method is applied to a fusion positioning system, and the method comprises: In response to a target object appearing in a preset collection area of an image collection unit in the fusion positioning system, radar data and image data corresponding to the target object collected by the image collection unit are acquired; Based on initial space synchronization parameters of the fusion positioning system, the radar data are subjected to space conversion to obtain a radar fitting track of the target object in an image coordinate system; The radar fitting track and a visual fitting track of the image data in the image coordinate system are compared to obtain a track comparison result; In response to the track comparison result not satisfying a preset condition, the initial space synchronization parameters are modified to obtain modified space synchronization parameters of the fusion positioning system; Based on the modified space synchronization parameters, the radar data and the image data are fused to obtain fusion positioning information of the target object, so as to realize fusion positioning of the target object.

2. The fusion positioning method of claim 1, wherein, The radar data at least include multiple frames of radar point cloud data of the target object in a radar coordinate system and radar time stamps corresponding to each frame of radar point cloud data, and the image data at least include multiple frames of visual images of the target object in a camera coordinate system and visual time stamps corresponding to each frame of visual image; The method further comprises: Based on the radar time stamp corresponding to each frame of radar point cloud data, the multiple frames of visual images are synchronized to obtain multiple frames of visual synchronization images corresponding to the multiple radar time stamps; Correspondingly, the comparison of the radar fitting track and the visual fitting track of the image data in the image coordinate system to obtain the track comparison result comprises: The comparison of the radar fitting track and the visual fitting track of the multiple frames of visual synchronization images in the image coordinate system to obtain the track comparison result.

3. The fusion positioning method of claim 2, wherein, The space conversion of the radar data based on the initial space synchronization parameters of the fusion positioning system to obtain the radar fitting track of the target object in the image coordinate system comprises: Based on the time sequence of the radar time stamps, starting from the radar point cloud data corresponding to the first radar time stamp, multiple three-dimensional data points corresponding to the target object in the radar point cloud data are projected onto the image coordinate system based on the initial space synchronization parameters to obtain multiple two-dimensional projection points; The circumscribed rectangle corresponding to the multiple two-dimensional projection points is determined as a radar detection box of the target object corresponding to the first radar time stamp, and a radar detection box of the target object corresponding to each radar time stamp is obtained; Based on the time sequence of the radar time stamps and the radar detection box of the target object corresponding to each radar time stamp, multiple radar tracking information of the target object in the image coordinate system is obtained; The multiple radar tracking information is subjected to track fitting to obtain the radar fitting track of the target object in the image coordinate system.

4. The fusion positioning method of claim 3, wherein, The initial space synchronization parameters at least include an extrinsic parameter, and the extrinsic parameter comprises a rotation matrix and a translation matrix corresponding to a preset rotation center point. Based on the initial spatial synchronization parameters, multiple three-dimensional data points corresponding to the target object in the radar point cloud data are projected onto the image coordinate system to obtain multiple two-dimensional projection points, including: Based on the rotation matrix, the radar point cloud data is rotated around the rotation center point to obtain the rotated radar point cloud data. Based on the translation matrix, the rotated radar point cloud data is translated to obtain the transformed radar point cloud data. The converted radar point cloud data is projected onto the image coordinate system to obtain multiple two-dimensional projection points corresponding to the target object.

5. The fusion positioning method of claim 2, wherein, The initial spatial synchronization parameters include at least intrinsic parameters, which include an intrinsic parameter matrix and an extrinsic parameter matrix. The method further includes: Based on the intrinsic parameter matrix and the extrinsic parameter matrix, an affine transformation is performed on the multi-frame visual synchronization image in the camera coordinate system to obtain the multi-frame visual transformation image of the multi-frame visual synchronization image in the image coordinate system at each radar timestamp. Based on the multi-frame visual transformation images, multiple visual detection boxes of the target object are determined on the image coordinate system; Based on the time sequence of the radar timestamps and the multiple visual detection boxes, the visual tracking information of the target object in the image coordinate system is obtained; The visual tracking information is fitted to a trajectory to obtain the radar-fitted trajectory of the target object in the image coordinate system.

6. The method of claim 1 to 5, wherein, The comparison of the radar fitted trajectory and the visual fitted trajectory of the image data in the image coordinate system to obtain the trajectory comparison result includes: Based on the radar fitting trajectory and the visual fitting trajectory in the image coordinate system, a plurality of second radar coordinates and a plurality of second visual coordinates corresponding to the radar fitting trajectory and the visual fitting trajectory at a plurality of first coordinates are determined respectively. By comparing the second radar coordinate and the second visual coordinate corresponding to each first coordinate, the second coordinate difference between the radar fitted trajectory and the visual fitted trajectory at each first coordinate is obtained. Based on each second coordinate difference and the number of first coordinates, the trajectory error between the radar fitted trajectory and the visual fitted trajectory is determined; The trajectory error is determined as the trajectory comparison result.

7. The method of claim 1 to 5, wherein, The step of correcting the initial spatial synchronization parameters to obtain the corrected spatial synchronization parameters of the fused positioning system includes: The radar fitting trajectory is rotated and translated until the trajectory error between the adjusted radar fitting trajectory and the visual fitting trajectory is less than the error threshold. The adjusted radar fitting trajectory is then determined as the target radar fitting trajectory of the target object. Extract the second radar coordinates corresponding to multiple first coordinates from the target radar fitted trajectory; Based on multiple second radar coordinates, the extrinsic parameters in the initial spatial synchronization parameters are corrected to obtain the corrected extrinsic parameters. The intrinsic parameters in the modified extrinsic parameters and the initial spatial synchronization parameters are determined as the modified spatial synchronization parameters of the fusion positioning system.

8. The method of claim 1 to 5, wherein, The initial space synchronization parameter is corrected to obtain a corrected space synchronization parameter of the fusion positioning system, including: The rotation matrix and the translation matrix in the initial space synchronization parameter are adjusted multiple times, and after each adjustment, an adjusted rotation matrix, an adjusted translation matrix and a corresponding adjusted radar fitted track are obtained; Each adjusted radar fitted track and the visual fitted track are matched respectively to obtain multiple matching degrees; The adjusted rotation matrix and the adjusted translation matrix corresponding to the adjusted radar fitted track with the highest matching degree are determined as the corrected extrinsic parameter; The corrected extrinsic parameter and the intrinsic parameter in the initial space synchronization parameter are determined as the corrected space synchronization parameter of the fusion positioning system.

9. The method of claim 1 to 5, wherein, The radar data further includes size information of the target object, and the image data further includes semantic information of the target object; The radar data and the image data are fused based on the corrected space synchronization parameter to obtain fusion positioning information of the target object, including: The radar data are spatially converted based on the corrected space synchronization parameter to obtain a corrected radar fitted track of the target object in an image coordinate system; In response to a track comparison result between the corrected radar fitted track and the visual fitted track satisfying a preset condition, the size information and the semantic information are fused to obtain the fusion positioning information.

10. A fusion positioning system characterized by, The fusion positioning system at least includes an image acquisition unit and a processor; wherein: The image acquisition unit is configured to acquire radar data and image data corresponding to a target object in response to the target object appearing in a preset acquisition region; The processor is configured to acquire the radar data and the image data corresponding to the target object acquired by the image acquisition unit, spatially convert the radar point cloud data based on an initial space synchronization parameter of the fusion positioning system to obtain a radar fitted track of the target object in an image coordinate system, compare the radar fitted track and a visual fitted track of the image data in the image coordinate system to obtain a track comparison result, and in response to the track comparison result not satisfying a preset condition, correct the initial space synchronization parameter to obtain a corrected space synchronization parameter of the fusion positioning system, so as to fuse the target object through the corrected space synchronization parameter.

Images