Data calibration method, device, electronic device and storage medium

Abstract

The present application relates to the field of intelligent transportation technology, and in particular to a data calibration method, device, electronic device and storage medium for improving the accuracy and efficiency of radar and video data calibration. The method includes: performing target detection on each frame of radar data and video data of a specified scene to obtain the position information of the target appearing in each frame; for each target, according to the error between the position information of the target in each historical frame and the estimated position information, and the position information of each target in the current frame, respectively obtain the first target trajectory and the second target trajectory of the target under the radar data and video data; the first feature points selected from the first target trajectories of different targets are formed into a first feature point group, and then formed into a feature point pair with the second feature points on the corresponding second target trajectory; according to the position mapping relationship of each feature point pair, the calibration parameters to be solved are optimized to obtain the calibration parameters of the radar data and video data.

Description

Background Art

[0002] In the field of security, surveillance technology based on radar and video all-in-one devices (referred to as: radar vision equipment) is gaining more and more attention. By combining the radar's all-weather working ability and long-distance detection advantages with the video sensor's high-precision target recognition ability, more comprehensive and accurate target detection and tracking can be achieved. The radar vision equipment needs to perform joint calibration of the millimeter-wave radar and the camera, that is, determine the mapping relationship between the radar coordinate system and the image coordinate system, and effectively integrate the radar data and video data.

[0003] In the related art, manual calibration methods are often used for calibration. This method is cumbersome and requires manual adjustment of multiple parameters to ensure spatial and temporal synchronization between radar and video sensors. This not only increases the workload, but is also prone to errors, which may lead to inconsistency and non-repeatability of calibration results. In addition, when faced with complex and changing environments, this method is difficult to adapt quickly, thus affecting the real-time and stability of the system.

[0004] In summary, how to improve the accuracy and efficiency of radar and video data calibration is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a data calibration method, device, electronic device and storage medium to improve the accuracy and efficiency of radar and video data calibration.

[0006] A data calibration method provided in an embodiment of the present application includes:

[0007] Performing target detection on each frame of the collected data of the specified scene to obtain the position information of the target appearing in each frame of the specified scene; the collected data includes radar data and video data;

[0008] For each target, according to the error between the position information of the target in each historical frame and the estimated position information, and the position information of each target in the current frame, respectively obtain a first target trajectory of the target under the radar data and a second target trajectory of the target under the video data;

[0009] After forming a first feature point group with first feature points selected from first target trajectories of different targets, forming a feature point pair with each first feature point in the first feature point group and a second feature point on a corresponding second target trajectory;

[0010] According to the position mapping relationship between the first feature point and the second feature point in each feature point pair, the calibration parameters to be solved are optimized to obtain the calibration parameters between the radar data and the video data.

[0011] An embodiment of the present application provides a data calibration device, comprising:

[0012] A target detection unit, used to perform target detection on each frame of the collected data of the specified scene, and obtain the position information of the target appearing in each frame of the specified scene; the collected data includes radar data and video data;

[0013] A trajectory determination unit is used to obtain, for each target, a first target trajectory of the target under radar data and a second target trajectory of the target under video data according to an error between the position information of the target in each historical frame and the estimated position information, and the position information of each target in a current frame;

[0014] A feature extraction unit, configured to form a first feature point group from first feature points selected from first target trajectories of different targets, and then form a feature point pair from each first feature point in the first feature point group and a second feature point on a corresponding second target trajectory;

[0015] The calibration unit is used to optimize the calibration parameters to be solved according to the position mapping relationship between the first feature point and the second feature point in each feature point pair, so as to obtain the calibration parameters between the radar data and the video data.

[0016] Optionally, the calibration unit is specifically used for:

[0017] For each calibration area in the specified scene, perform the following operations respectively:

[0018] Constructing a calibration optimization target according to a position mapping relationship between a first feature point and a second feature point in each feature point pair in the calibration area; wherein each calibration area is obtained by dividing the specified scene along a collection direction corresponding to the collection data; and the calibration optimization target represents: a difference between a feature point predicted by the position mapping relationship and a real feature point;

[0019] Based on the calibration optimization target, the calibration parameters to be solved are optimized to obtain the calibration parameters corresponding to the calibration area.

[0020] Optionally, the position information obtained by performing target detection based on the collected data is the actual position information; each of the estimated position information is: the position information of the trajectory point represented by the trajectory equation to be solved; the trajectory determination unit is specifically used for:

[0021] Constructing a trajectory optimization target according to the error between the actual position information of the target in each historical frame and the estimated position information of the corresponding trajectory point;

[0022] Based on the trajectory optimization target, after optimizing the trajectory equation to be solved, the estimated position information of the target in the current frame is predicted;

[0023] After target matching is performed based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame, the target trajectory of the target under the collected data of each frame is obtained according to the target matching result and the actual position information of the target in each historical frame; the target trajectory includes: a first target trajectory of the target under radar data, and a second target trajectory of the target under video data.

[0024] Optionally, the position information includes a plurality of position parameters, each position parameter corresponds to a trajectory equation to be solved; and the trajectory determination unit is specifically used for:

[0025] For each position parameter, determining a trajectory optimization target corresponding to the position parameter based on an error between the actual position parameter of the target in each historical frame and the estimated position parameter of the corresponding trajectory point;

[0026] For each position parameter, based on the trajectory optimization target, the corresponding trajectory equation to be solved is optimized to obtain the optimized trajectory equation of the position parameter;

[0027] Based on the optimized trajectory equations of the position parameters, the estimated position information of the target in the current frame is predicted.

[0028] Optionally, the trajectory determination unit is specifically used to:

[0029] Based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame, respectively determining the matching error between the target and each target;

[0030] Determine, among the matching errors, a target matching error that belongs to a preset matching error range and has a minimum error value;

[0031] The actual position information used to determine the target matching error is used as the target matching result of the target.

[0032] Optionally, the trajectory determination unit is further used for:

[0033] If the matching errors do not fall within the preset matching error range, then analyzing whether the target trajectory frame number meets the threshold; the trajectory frame number is: the frame number of the target acquisition data in each frame of acquisition data;

[0034] If the preset threshold is not met, it is determined that the target is lost.

[0035] Optionally, the feature extraction unit is specifically used for:

[0036] For each calibration area in the specified scene, perform the following operations respectively:

[0037] The first feature points selected from the first target trajectories of different targets, located in the calibration area and capable of constructing a triangle, are formed into a first feature point group; wherein each calibration area is obtained by dividing the designated scene along the collection direction corresponding to the collection data;

[0038] For each first feature point group, each first feature point in the first feature point group and a second feature point on a corresponding second target trajectory form a feature point pair.

[0039] Optionally, for each first feature point group, before each first feature point in the first feature point group and a second feature point on a corresponding second target trajectory are combined into a feature point pair, the feature extraction unit is further configured to:

[0040] For each first feature point group, construct a feature triangle using the first feature points in the first feature point group;

[0041] According to the similarity between the feature triangles, each candidate triangle group is determined from the constructed feature triangles; each candidate triangle group includes: each feature triangle whose similarity meets a preset threshold;

[0042] For each candidate triangle group, only the first feature point group corresponding to one feature triangle in the candidate triangle group is retained.

[0043] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned data calibration methods.

[0044] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the above-mentioned data calibration methods.

[0045] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned data calibration methods.

[0046] The beneficial effects of this application are as follows:

[0047] The embodiment of the present application provides a data calibration method, device, electronic device and storage medium. In the specific implementation, first, by performing target detection on each frame of radar and video data, the position information of the target is obtained, which provides an accurate data basis for the calibration process, and at the same time ensures that the calculation of the calibration parameters depends on the accurate target position information, avoiding the defects of relying on manual input or roughly estimating the position in the related art, ensuring the accuracy of the calibration parameter calculation, and reducing the possibility of error accumulation. Then, according to the position information and estimated position of the target in the historical frame and the current frame, the target trajectory under the radar and video data is constructed respectively. Compared with the calibration relying on a single frame or discontinuous data in the related art, the present application effectively tracks the target through a continuous trajectory, thereby improving data consistency and enhancing the stability of the calibration process. Subsequently, feature points are selected from the trajectories of different targets, and by matching these feature points, feature point pairs are formed to ensure that the data in the calibration process is more accurate, remove redundant information, improve calculation efficiency, avoid interference from meaningless data, and improve the accuracy and speed of calibration. Finally, by analyzing the position mapping relationship of each pair of feature points, the calibration parameters are optimized to ensure the precise mapping of the spatial relationship between radar and video data, thereby improving the accuracy and efficiency of calibration. The optimization process effectively overcomes the problems of poor adaptability and low real-time performance of related technologies in complex environments, enabling radar and video data to be calibrated efficiently and accurately in a changing environment.

[0048] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0050] Figure 1A A schematic diagram of radar and video target detection error mapping provided in an embodiment of the present application;

[0051] Figure 1B A schematic diagram of correct mapping of radar and video target detection provided in an embodiment of the present application;

[0052] Figure 2 A flowchart of a data calibration method provided for an application embodiment;

[0053] Figure 3 A schematic diagram of target detection using video and radar provided in an embodiment of the present application;

[0054] Figure 4 A flowchart for obtaining a target trajectory provided in an embodiment of the present application;

[0055] Figure 5 A flowchart of a video and radar acquisition of target trajectory status provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a target trajectory provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of the division of a calibration area provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of constructing a characteristic triangle provided in an embodiment of the present application;

[0059] Fig. 9 A flow chart of constructing a characteristic triangle provided in an embodiment of the present application;

[0060] Fig.10 A flow chart for determining calibration parameters provided in an embodiment of the present application;

[0061] Fig.11 An overall flow chart of a data calibration method provided in an embodiment of the present application;

[0062] Fig.12 A schematic diagram of the structure of a data calibration device in an embodiment of the present application;

[0063] Fig.13 A schematic diagram of the hardware structure of an electronic device to which an embodiment of the present application is applied. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0065] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0066] With the rapid development of autonomous driving technology, drone navigation, and intelligent monitoring systems, the fusion application of radar and video sensors has become increasingly important. Radar and video sensors each have unique advantages: radar can work stably in all weather conditions and provide accurate distance and speed information; video sensors can provide rich visual information for target recognition and classification. However, to fully realize the potential of these sensors, accurate calibration is necessary to ensure that the data between them can be effectively fused.

[0067] It should be noted that the target detected by a normal millimeter-wave radar is a clustered area, and the radar target information obtained is a point in the radar coordinate system. The video target is a rectangular area in the image coordinate system. Through the calibration algorithm, the target point in the millimeter-wave radar coordinate system can be mapped to the pixel point in the image coordinate system.

[0068] In traffic monitoring, the fusion of radar and video sensors can provide more comprehensive traffic information, including vehicle speed, location, type, etc. Accurate and stable radar and video fusion results are the cornerstone for subsequent judgment of traffic road conditions.

[0069] See also Figure 1A FIG. 1 is a schematic diagram of radar and video target detection error mapping provided by an embodiment of the present application. Figure 1A In the scene shown, 1, 2, 3, and 4 represent four vehicle targets. The video detection target is the solid rectangular box in the figure. After the initial conversion, the radar target in the radar coordinate system can be mapped to the image coordinate system to form a dotted rectangular box. The existence of a solid box indicates that the vehicle target is detected by the visual sensor, and the existence of a dotted box indicates that the vehicle target is detected by the radar sensor.

[0070] Before fusion, it is expected that the radar detection results and visual detection of each target can be synchronized. Figure 1A After the radar detection results are mapped to the image, there is a large deviation from the image detection results. Figure 1A The radar result of target 1 is correlated with the visual result of target 2. Figure 1A The radar results of the 3 targets cannot be associated with the visual results, which leads to incorrect fusion results.

[0071] See also Figure 1B As shown, a schematic diagram of correct mapping of radar and video target detection provided by an embodiment of the present application, the scene and Figure 1A The same, except that Figure 1BThe correct mapping results in can synchronize the radar results with the visual results and obtain better fusion results. In this correct mapping process, the target positions of the radar data and the video data are accurately aligned, ensuring the consistency of the detection results of each target between the two sensors. Correct fusion can avoid Figure 1A It can eliminate the mismatching and misalignment problems that occur in the image processing, thereby improving the accuracy of target detection and providing reliable data support for subsequent target tracking and prediction.

[0072] When radar and video are combined for target detection, the relevant technology usually adopts manual calibration, which may lead to inconsistent and non-repeatable calibration results. For example, during the manual calibration process, the operator needs to manually adjust the mapping relationship between radar and video data. If the environment changes (such as vehicle speed changes, lighting changes, occlusion, etc.), the calibration results may be different each time, which may make it impossible to guarantee the accuracy and consistency of the calibration. Figure 1A As shown, after manual calibration, there may be obvious deviations in the mapping between radar targets and video targets, resulting in misalignment between the calibrated dotted box (radar detection result) and the solid box (video detection result).

[0073] In addition, when faced with complex and changing environments, manual calibration methods are difficult to quickly adapt to environmental changes, thus affecting the real-time performance and stability of the system. Due to dynamic changes in the environment (such as changes in illumination, changes in the speed of the target, etc.), the calibration parameters cannot be adjusted in time after manual calibration, which causes large errors in the fusion of radar and video data. In this case, it is impossible to effectively cope with the rapid changes in the environment, resulting in unsatisfactory fusion effects and affecting the accuracy of subsequent target tracking.

[0074] Based on this, an embodiment of the present application provides a data calibration method for improving the accuracy and efficiency of radar and video data calibration. For example, in a traffic scene, the data calibration method provided by the present application can be used to perform target detection on each frame of the data collected in the specified scene to obtain the position information of the target in each frame; obtain the target trajectory under the radar data and the video data respectively according to the historical position information of the target and the estimated trajectory error; select and match feature points, and optimize the calibration parameters based on the feature point position mapping relationship, so as to achieve accurate calibration between radar and video data. This method does not require human intervention and additional sensor assistance, effectively improves the accuracy of data calibration, and improves the accuracy of subsequent target tracking.

[0075] The following describes the data calibration method provided by the exemplary embodiment of the present application in combination with the design concept described above and with reference to the accompanying drawings. It should be noted that the above design concept is only shown to facilitate understanding of the spirit and principles of the present application, and the implementation method of the present application is not limited in this regard.

[0076] It should be noted that the data calibration method in each embodiment of the present application can be executed by an electronic device, which may be a terminal device or a server. The method can be executed by the terminal device or the server alone, or jointly by the terminal device and the server.

[0077] Among them, terminal devices include but are not limited to mobile phones, tablets, laptops, desktop computers, monitoring equipment such as radar and video all-in-one machines, etc.; a client related to data calibration can be installed on the terminal device, and the client can be software, or a web page, small program, etc.; the server is a background server corresponding to the software or web page, small program, etc., or a server specifically used for data calibration, and this application does not make specific limitations.

[0078] See also Figure 2 As shown, it is an implementation flow chart of a data calibration method provided in an embodiment of the present application. Taking the server as the execution subject as an example, the specific implementation process of the method is as follows S21~S24:

[0079] S21: performing target detection on each frame of the acquired data of the specified scene to obtain the position information of the target appearing in each frame of the specified scene; the acquired data includes radar data and video data.

[0080] Among them, the designated scene refers to a complex or dynamic environment with multiple targets moving, which is a specific scene that requires data calibration, such as a dynamic environment with multiple targets such as a traffic section or a parking lot captured by a radar-based integrated device.

[0081] Collected data refers to data collected in a specified scene by a radar-vision integrated device or other sensing equipment, which contains multiple continuous frames. Each frame of data includes two categories: radar data and video data. Radar data transmits and receives electromagnetic waves through radar sensors, captures the position, speed and other information of objects in the scene, and forms point cloud data; video data obtains the image information of the target through the camera, and extracts the shape, position and other features of the target through image processing technology. The combination of radar and video data can provide comprehensive perception of the target and help achieve higher-precision target detection and calibration.

[0082] Target detection refers to extracting target features (such as shape, outline, motion state, etc.) from radar data and video data respectively through radar point cloud data processing and video image analysis technology, and obtaining the target position through a target detection algorithm (which can be an image detection algorithm based on deep learning or a clustering and classification algorithm based on radar point cloud).

[0083] In the embodiments of the present application, the target refers to various objects or entities that need to be detected and tracked in a specified scene, including but not limited to vehicles, ships, aircraft, animals, etc. The specific target type will be flexibly adjusted according to the actual application scenario. For example, in a traffic monitoring scenario, vehicles are usually the main detection targets; in an airport monitoring scenario, aircraft may be the key detection object; in a smart parking lot application, the occupancy status of parking spaces and the entry and exit of vehicles can also be tracked and calibrated as targets. This article does not specifically limit the target type, so the target can be any object that needs to be detected, identified, and tracked in a specific environment.

[0084] In an embodiment of the present application, by performing target detection on each frame of collected data (including radar and video data) in a specified scenario, the target appearing in each frame and its position information in the corresponding frame can be extracted.

[0085] In an optional embodiment, the location information of the target includes at least spatial parameters, which are used to describe the spatial information of the target in the target detection area. For the location information of the target in the radar coordinate system corresponding to the radar data, it can be the position of the target in the radar coordinate system, such as the (x, y) set of the target point cloud in the radar coordinate system, where x represents the lateral position of the target in the radar coordinate system, and y represents the longitudinal position of the target in the radar coordinate system); the location information of the target in the video coordinate system corresponding to the video data can be the set of corresponding pixel points (u, v) in the video image, where u represents the pixel position of the target in the horizontal direction of the image, and v represents the pixel position of the target in the vertical direction of the image.

[0086] It should be noted here that the target location information under the video data is actually an area composed of several pixels, which is the part of the target that is identified in the image. However, when specifically determining the location of the target, you can choose a pixel as a representative to simplify the positioning result instead of directly using all the pixels. In contrast, the radar result is usually a single point, which represents the clear location of the target.

[0087] In another optional implementation, the position information of the target may also include motion parameters, which are used to describe information such as the speed and direction of movement of the target. These motion parameters can provide support for subsequent trajectory estimation and target prediction. Motion parameters are particularly suitable for dynamic scenes. For example, in a complex traffic environment, information such as the speed and direction of movement of the target can accurately predict the future motion trajectory of the target, further improving the accuracy of the calibration process.

[0088] S22: For each target, according to the error between the position information of the target in each historical frame and the estimated position information, and the position information of each target in the current frame, obtain the first target trajectory of the target under the radar data and the second target trajectory of the target under the video data.

[0089] In the embodiment of the present application, the target trajectory may be the first target trajectory (under the radar coordinate system) or the second target trajectory (under the video coordinate system). The above processing may be performed for different types of collected data respectively.

[0090] In an optional implementation, the target trajectory is mainly obtained through trajectory points, and the determination of the trajectory points is completed by combining the position information of the historical frame, the estimated position information and the actual position information of the current frame.

[0091] Specifically, first, the estimated position information of the current frame is obtained based on the error between the position information of the target in the historical frame and the estimated position information. Then, the estimated position information is matched with the target position information appearing in the current frame to obtain the actual position information of the target in the current frame. By adding the position information of the current frame to the historical frame and updating the current frame to the next frame, the position of the target can continue to be updated in subsequent frames. Finally, through this process, continuous target position information is summarized to construct the target trajectory. For radar data and video data, their target trajectories can be represented as the first target trajectory (trajectory in the radar coordinate system) and the second target trajectory (trajectory in the video coordinate system), respectively. The key to determining the trajectory point is to obtain the accurate target position by matching historical data and real-time data, thereby ensuring the accurate construction of the trajectory.

[0092] In the embodiments of the present application, the estimated position information is mainly determined by parameter estimation, and the position information (actual position information) is determined by trajectory matching. These two processes are briefly introduced below.

[0093] See also Figure 3 As shown, it is a schematic diagram of target detection using video and radar provided in an embodiment of the present application.

[0094] exist Figure 3 In the data, radar and video can both detect the number and location of vehicles. Therefore, a unified description can be established for radar data and video data to describe the location information. Taking vehicle A as an example, the location information of vehicle A can be:

[0095] Formula 1

[0096] Take radar as an example. It can refer to the spatial parameter x of radar data, It can refer to the spatial parameter y of the radar data. In the current frame, the position information of each vehicle detected in the current frame is:

[0097] Formula 2

[0098] Among them, i represents the i-th vehicle detected in the current frame, Represents the position information of the i-th vehicle in the current frame.

[0099] Historical location information detected based on historical frames The trajectory parameter equation can be estimated by the parameter estimation method: , the estimated location information is , the mapping relationship is:

[0100] Formula 3

[0101] in, It represents the estimated position information of the target vehicle (which can be vehicle A in this case) in the jth frame among the first M frames, and F is the trajectory parameter equation of the target vehicle.

[0102] Through the above process, the process of trajectory parameter estimation is briefly introduced. By analyzing the historical position information, the position of the target in the current frame can be estimated, thereby improving the accuracy of target tracking.

[0103] For trajectory matching, it is mainly through trajectory parameter matching function , you can build the current frame detection target result And historical trajectory results The matching relationship of , then the current trajectory The calculation of is as follows:

[0104] Formula 4

[0105] In an embodiment of the present application, the trajectory parameter matching function is used to construct a matching relationship between the target detected in the current frame and the historical trajectory results. The current trajectory can be expressed by the above formula 4, and the parameters in formula 4 are updated according to the matching relationship between the trajectory prediction results of the historical frame and the target detection results of the current frame. Specifically, the trajectory matching function compares the position information of the target in the current frame with the matching degree of the historical trajectory obtained by parameter estimation, thereby determining whether the target is part of the historical trajectory, and updating the trajectory state of the target accordingly.

[0106] Through trajectory matching, the target can be accurately matched in each frame of data and a continuous target trajectory can be established between different frames, thereby improving the accuracy of radar and video data calibration and ensuring the accurate fusion of multi-sensor data.

[0107] In order to further implement the process of parameter trajectory and trajectory matching, in an optional implementation, more specific trajectory estimation and target matching can be achieved through the following steps.

[0108] See also Figure 4 As shown, it is a flowchart of obtaining a target trajectory provided by an embodiment of the present application, and the specific steps are as follows S400~S403:

[0109] S400: The position information obtained by performing target detection based on the collected data is the actual position information; each estimated position information is: the position information of the trajectory point represented by the trajectory equation to be solved.

[0110] In the embodiment of the present application, the estimated position information is determined by constructing parameter equations to be solved for radar and video trajectories respectively. For the position information obtained by target detection of radar data or video data, the position information includes multiple position parameters, each of which corresponds to a trajectory equation to be solved.

[0111] In an optional implementation, each target corresponds to two trajectory equations to be solved. The following example is used to illustrate:

[0112] For target A in the radar coordinate system, the trajectory equation to be solved for the horizontal coordinate x of target A is as follows:

[0113] Formula 5

[0114] in, Represents the position information of target A at the horizontal coordinate x in the radar coordinate system at time frame t, , , , It is the parameter to be solved of the trajectory equation of the horizontal coordinate x in the radar coordinate system of target A.

[0115] For target A in the radar coordinate system, the trajectory equation to be solved for the ordinate y of target A is as follows:

[0116] Formula 6

[0117] in, It represents the position information of target A at the vertical coordinate y in the radar coordinate system at time frame t. , , , It is the parameter to be solved of the trajectory equation of the ordinate y in the radar coordinate system of target A.

[0118] In summary, for target A in the radar coordinate system, the estimated position information of target A obtained in the first t frames can be expressed as:

[0119] Formula 7

[0120] in, Represents the position information in the radar coordinate system, Represents the estimated radar coordinate position of target A at the i-th frame.

[0121] For target A in the video coordinates, the horizontal pixel direction of target A u The trajectory equation to be solved is as follows:

[0122] Formula 8

[0123] in, Represents the horizontal pixel direction of target A in the video coordinate system at time frame t u location information, , , , is the horizontal pixel direction of target A in the video coordinate system u The parameters to be solved of the trajectory equation to be solved.

[0124] For target A in the video coordinates, the vertical pixel direction of target A v The trajectory equation to be solved is as follows:

[0125] Formula 9

[0126] in, Represents the position information of target A in the vertical pixel direction v in the video coordinate system at time frame t, , , , It is the parameter to be solved of the trajectory equation to be solved in the vertical pixel direction v in the video coordinate system of target A.

[0127] In summary, for target A in the video coordinate system, the estimated position information of target A obtained in the previous t frames can be expressed as:

[0128] Formula 10

[0129] in, Represents the position information estimated by the trajectory equation of two position parameters in the video coordinate system, Represents the estimated video coordinate position of target A at the i-th frame.

[0130] After obtaining the estimated position information, the trajectory equation to be solved can be optimized frame by frame through the estimated position information and the actual position information while performing target matching to obtain the actual position information, so as to obtain the trajectory equation corresponding to each position parameter of the target. The process of optimizing the trajectory equation to be solved is described below, such as S401 to S403 below:

[0131] S401: Constructing a trajectory optimization target according to the error between the actual position information of the target in each historical frame and the estimated position information of the corresponding trajectory point.

[0132] In an optional implementation, the position information includes multiple position parameters, each position parameter corresponds to a trajectory equation to be solved, such as shown in Formula 5 and Formula 6 in the radar coordinate system or in Formula 8 and Formula 9 in the video coordinate system.

[0133] Optionally, for each position parameter, a trajectory optimization target corresponding to the position parameter is determined based on an error between an actual position parameter of the target in each historical frame and an estimated position parameter of a corresponding trajectory point.

[0134] In an optional implementation, the detection targets of the current radar and video are obtained respectively. Assume that the actual position information of the kth target in the i-th frame among the multiple targets detected by the video target is , and the actual position information of the i-th frame The residuals (i.e. errors) can be constructed separately:

[0135] Formula 11

[0136] Formula 12

[0137] Among them, i represents the frame number, represents the actual horizontal pixel position of the kth target in the i-th frame, represents the estimated horizontal pixel position of the kth target in the i-th frame, Represents the residual (error) of the k-th target in the horizontal pixel direction of the i-th frame, that is, the interpolation value between the actual horizontal pixel position and the estimated horizontal pixel position; represents the actual vertical pixel position of the kth target in the i-th frame, represents the estimated vertical pixel position of the kth target in the i-th frame, It represents the residual (error) of the k-th target in the vertical pixel direction of the i-th frame, that is, the difference between the actual vertical pixel position and the estimated vertical pixel position.

[0138] Assume that the jth frame of the kth target among multiple targets detected by the radar is , and the actual position information of the jth frame The residuals can be constructed separately:

[0139] Formula 13

[0140] Formula 14

[0141] Among them, j represents the frame number, represents the actual horizontal coordinate position of the kth target in the jth frame, represents the estimated horizontal coordinate position of the kth target in the jth frame, Represents the residual (error) of the k-th target in the horizontal direction of the j-th frame, that is, the difference between the actual horizontal coordinate position and the estimated horizontal coordinate position; represents the actual ordinate position of the kth target in the jth frame, represents the estimated ordinate position of the kth target in the jth frame, It represents the residual (error) of the k-th target in the ordinate direction of the j-th frame, that is, the difference between the actual ordinate position and the estimated ordinate position.

[0142] According to the above residual, the trajectory optimization objective can be constructed for the video as:

[0143] Formula 15

[0144] Formula 16

[0145] Among them, i represents the frame number, and the value range is 1 to n, where n represents the total number of historical frames for target detection in the video coordinate system.

[0146] In the video coordinate system, the purpose of the trajectory optimization target in the horizontal pixel direction shown in Formula 15 is to minimize the residual sum of squares of all frames in the horizontal direction, and the purpose of the trajectory optimization target in the vertical pixel direction shown in Formula 16 is to minimize the residual sum of squares of all frames in the vertical direction.

[0147] The trajectory optimization objective for the radar is:

[0148] Formula 17

[0149] Formula 18

[0150] Among them, i represents the frame number, and the value range is 1 to n, where n represents the total number of historical frames for target detection in the radar coordinate system.

[0151] In the radar coordinate system, the purpose of the trajectory optimization target on the horizontal axis shown in Formula 17 is to minimize the residual sum of squares of all frames on the horizontal axis, and the purpose of the trajectory optimization target on the vertical axis shown in Formula 18 is to minimize the residual sum of squares of all frames on the vertical axis.

[0152] Through the above process, the trajectory optimization target is constructed based on the error between the actual position information of the target in each historical frame and the estimated position information of the corresponding trajectory point. Specifically, for video and radar data, we calculate the residuals (errors) in the horizontal and vertical directions respectively, and construct the trajectory optimization target based on these residuals. The optimization target aims to minimize the sum of squared residuals of all frames, thereby improving the accuracy of trajectory prediction.

[0153] S402: Based on the trajectory optimization target, after optimizing the trajectory equation to be solved, the estimated position information of the target in the current frame is predicted.

[0154] In an embodiment of the present application, the position information includes multiple position parameters, each position parameter corresponds to a trajectory equation to be solved.

[0155] Optionally, for each position parameter, based on the trajectory optimization target, the corresponding trajectory equation to be solved is optimized to obtain the optimized trajectory equation of the position parameter; based on the optimized trajectory equations of each position parameter, the estimated position information of the target in the current frame is predicted.

[0156] Trajectory optimization based on each position parameter can improve the accuracy of the target's estimated position information in the current frame. For each position parameter, each position parameter can be optimized separately to avoid the mutual influence of multiple parameters, thereby improving the overall prediction accuracy and ensuring accurate prediction of the target trajectory in multiple dimensions. This processing method of position parameters helps to more finely adjust and optimize the calculation results of each trajectory point, making the final trajectory estimation more accurate and stable.

[0157] In the embodiment of the present application, the above optimization process is iteratively optimized for each frame, and the frame-by-frame optimization process is explained with the following example.

[0158] Next, we will detect the target using radar data and obtain three targets, A, B, and C. The actual position information of these three targets is , , , taking the x of target A as an example, the process of optimizing frame by frame to obtain the estimated position x of target A in the current frame is explained.

[0159] When the current frame is the first frame, the actual position information of target A is X, and the trajectory equation to be solved is established as

[0160] Formula 19

[0161] For specific parameter introduction, please refer to formula 5, which will not be repeated here.

[0162] When the current frame is the first frame, since there is no historical frame information, the historical frame t=0, and at this time, an initial value can be assigned to the trajectory equation to be solved in the x direction of target A. In an optional implementation, the initial value can be set to zero, that is, it is assumed that target A is located at the origin of the coordinates in the first frame. In another optional implementation, the value can be assigned based on known position information, such as assuming that the position of target A in the first frame is 100 meters, or the target position inferred based on scene knowledge is set to a certain value, such as 50 meters. These initial values ​​provide a starting point for the trajectory equation, and then the trajectory equation will be continuously optimized as the target position is updated.

[0163] At this time, the actual position information of target A in the x direction is , the predicted location information is .

[0164] When the current frame is the second frame, the actual position information of the historical frame is the actual position information of the first frame. At this time, referring to formula 17, the trajectory optimization target is constructed as:

[0165] Formula 20

[0166] The trajectory equation to be solved is optimized by the trajectory optimization target. In the embodiment of the present application, the trajectory equation is adjusted by minimizing the trajectory optimization target so that the error is minimized. In an optional implementation, a least squares optimization algorithm is used to iteratively update the trajectory parameters until an optimal trajectory equation that can accurately describe the target motion is obtained. At this time, the optimized trajectory equation can be obtained when the current frame is the second frame. . Through the trajectory equation of the current frame as the second frame, the estimated position information of the current frame as the second frame is obtained .

[0167] It should be noted here that the optimization algorithm includes but is not limited to the least squares method. In other implementations, other optimization algorithms, such as Kalman filtering, particle filtering, gradient descent method, etc., can also be used for trajectory optimization. The selection of a specific optimization algorithm should be determined based on the actual application scenario, computing resources, and the motion characteristics of the target. This application does not specifically limit the optimization algorithm used.

[0168] Still taking the second frame as an example, first, three targets are still detected in the second frame according to the history, and the actual position information of the targets in the second frame are: , , ,At this time, the three detected targets do not correspond to the corresponding positions, because obtaining the actual position information of target A requires trajectory matching (also called target matching) to obtain the actual position information of target A in the current frame.

[0169] Optionally, the target matching process is achieved by:

[0170] Based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame, the matching errors between the target and each target are determined respectively; among each matching error, the target matching error that belongs to the preset matching error range and has the smallest error value is determined; and the actual position information used to determine the target matching error is used as the target matching result of the target.

[0171] Among them, the preset matching error range is a pre-set value or threshold, which can be a specific value or an interval, and is usually used to limit the error within an acceptable range, thereby helping to eliminate non-conforming matching results during the target matching process. For example, the preset matching error range can be set to ±5 pixels (in video data), which means that only matching results with target position information errors within ±5 pixels will be considered valid. If the error exceeds this range, the matching result will be eliminated. Similarly, for radar data, the error range can be set to ±0.1 meters (in the radar coordinate system), that is, the match will only be considered valid if the position error of the target in the radar coordinate system is less than 0.1 meters.

[0172] It should be noted here that the specific value range of the matching error in this application can be adjusted according to the characteristics of the target, the noise level of the data, and the application scenario. For example, when the target moves quickly or the signal noise is large, the error range can be appropriately relaxed; and when the target is stationary or the noise is low, a stricter matching error range can be set. This application does not make specific restrictions on this.

[0173] In an optional implementation, still taking the x direction of the target A in the second frame as an example, the estimated position information of the target A in the second frame is The actual position information of each target 1, 2, 3 detected in the current frame in the second frame , , The matching error between .

[0174] In an optional implementation, the matching error can be determined by using the Euclidean distance method, the weighted distance method, or a method based on machine learning. The specific method can be selected according to different application scenarios. This application does not impose specific restrictions on the method for calculating the matching error. The purpose of determining each matching error is to find the most appropriate matching relationship, thereby ensuring that the target can be correctly tracked and located between different frames. For example, the preset matching error range is a certain pixel distance or spatial error. By calculating the error between the estimated position information of target A and the actual position information of targets 1, 2, and 3, the target with the smallest matching degree is found to ensure the accuracy and reliability of the matching result.

[0175] Through the above process, accurate trajectory optimization can be achieved. Based on the error between the actual position information and the estimated position information of the target in the historical frame, the trajectory optimization target is constructed, thereby effectively optimizing the trajectory equation, reducing the error caused by the deviation of the initial trajectory equation, and improving the accuracy of the target trajectory. Based on the optimized trajectory equation, the estimated position information of the target in the current frame can be accurately predicted, providing a more accurate initial value for subsequent target matching and reducing the prediction error. Through the target matching process, target matching can be performed based on the estimated position information of the target in the current frame and the actual position information of each target, ensuring the accuracy of the matching results, and then constructing a more reliable target trajectory. Finally, through this process, the construction of the target trajectory can be optimized, the accuracy of the target matching can be improved, and the efficiency and accuracy of the matching results can be guaranteed.

[0176] Continuing with the second frame as an example, assume that the estimated position information of target A in the second frame is , the actual position information of targets 1, 2, and 3 detected in the current frame are , , By calculating the matching error between the estimated position information and the actual position information, it is assumed that the actual position information that best matches target A can be determined as , and update the actual position information of target A in the current frame as , the estimated position information of target A in the second frame is obtained as , the actual location information is .

[0177] What is needed here is that the trajectory parameter equation at this time may have a certain deviation, resulting in a large matching error between the estimated position information and the actual position information. This is because the initial trajectory equation of the target in the first frame may not fully and accurately reflect the real motion trajectory of the target. Therefore, in this case, although the error between the actual position and the estimated position of the target is adjusted by trajectory matching, the accuracy of the matching result may still be affected to a certain extent due to the deviation of the initial trajectory equation.

[0178] In an optional implementation, in addition to the matching method based on error minimization, other optimization algorithms, such as the Hungarian algorithm, can also be used to further improve the matching accuracy. The Hungarian algorithm effectively handles the multi-target matching problem, reduces errors through optimal matching, and optimizes the construction of the target trajectory. In addition, the matching accuracy can also be improved based on the particle filter algorithm. The particle filter can accurately estimate the target position in a nonlinear and complex dynamic environment by sampling a group of particles and gradually updating their weights. In this way, the initial trajectory deviation can be better dealt with and the accuracy of the matching results can be improved. Of course, other algorithms can also be used for target matching, and this application does not make specific restrictions on this.

[0179] When the current frame is the third frame, the actual position information of the historical frame is the actual position information of the first and second frames. At this time, the trajectory optimization target is:

[0180] Formula 21

[0181] Through the trajectory optimization target, the optimization algorithm is used to optimize the trajectory equation to be solved. The optimization algorithm refers to the introduction of the second frame, and this application will not repeat it. At this time, the optimized trajectory equation when the current frame is the third frame can be obtained. . Through the trajectory equation of the current frame being the third frame, the estimated position information of the current frame being the third frame is obtained .

[0182] The following process can refer to the above processing of the second frame, and similar optimization and estimation can be performed frame by frame. For example, based on the current frame being the third frame, further optimization is performed to the fourth frame, and so on, until all frames that need to be calculated are processed.

[0183] The number of frames of the current frame can be selected according to the actual situation, usually set to 60 frames, or adjusted according to specific needs. For example, the appropriate number of frames can be selected according to the frequency of target movement, the limitation of computing resources, or the real-time requirements. The purpose of selecting the appropriate number of frames is to ensure the accuracy of the trajectory parameters while avoiding the calculation being too complicated or time-consuming, so as to achieve a balance between real-time and accuracy.

[0184] S403: After target matching based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame, the target trajectory of the target under the collected data of each frame is obtained according to the target matching result and the actual position information of the target in each historical frame; the target trajectory includes: a first target trajectory of the target under the radar data, and a second target trajectory of the target under the video data.

[0185] In the embodiment of the present application, the purpose of target matching is to determine the matching relationship between the target and other targets by comparing the estimated position information of the target in the current frame with the actual position information, so as to accurately construct the motion trajectory of the target. This process can ensure that the position of the target is accurately tracked in multiple frames of data, and optimize the construction of the target trajectory by using the position information of the historical frames.

[0186] Optionally, the target matching process is achieved by:

[0187] Based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame, the matching errors between the target and each target are determined respectively; among each matching error, the target matching error that belongs to the preset matching error range and has the smallest error value is determined; and the actual position information used to determine the target matching error is used as the target matching result of the target.

[0188] For example, assume that in a certain video frame, the estimated position information of target A is (150, 200), that is, it is expected to appear at the pixel position (150, 200) of the image. In the current frame, there are multiple targets in the image, among which the actual position information of targets B, C, and D are (152, 202), (148, 198), and (155, 205), respectively. Assume that the preset matching error range is ±5 pixels. Then, we first calculate the error between target A and each target: The error between target A and target B is

[0189]

[0190] The error between target A and target C is 2.83 pixels (calculation process omitted, the same below), and the error between target A and target D is 7.07 pixels. Since the error between target A and target D exceeds the preset error range of ±5 pixels, it will be eliminated. The remaining matching errors between target A and target B and target C are all within the acceptable range, and the smallest error is the matching error between target A and target B (2.83 pixels). Therefore, the matching result of target A will select the actual position information of target B (152, 202) as the matching result.

[0191] Through the above process, the target matching can be realized based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame, and the matching error is calculated. According to the calculated matching error, the matching result with the smallest error value and in accordance with the preset error range is selected as the matching result of the target. In this way, the accurate matching of the target can be ensured, and the target trajectory can be effectively constructed. According to the target matching result and the actual position information in the historical frame, the target trajectory is further constructed, including the first target trajectory under the radar data and the second target trajectory under the video data.

[0192] Optionally, if each matching error does not fall within a preset matching error range, then the number of target trajectory frames is analyzed to determine whether it meets a threshold; the number of trajectory frames is: the number of frames of collected data in which the target appears in each frame of collected data;

[0193] If the preset threshold is not met, the target is determined to be lost.

[0194] The number of trajectory frames refers to the number of times the target successfully appears in each frame of the collected data, that is, the number of frames in which the target can be detected and successfully matched in a certain frame of data. Assuming that the preset threshold is 5 frames, that is, when the target is not successfully matched in 5 consecutive frames, it will be judged as lost.

[0195] In the embodiment of the present application, it is assumed that when the trajectory data of target A is processed, in 10 consecutive frames of data, target A is detected and successfully matched in 4 frames of data, but target A is not successfully detected in 6 frames of data. Then the trajectory frame number of target A is 4 frames, because it appears in 4 frames of data and is successfully matched. Since the trajectory frame number does not reach the preset 5-frame threshold, target A will be determined to be lost and tracking of the target will be stopped.

[0196] By introducing the judgment mechanism of trajectory frame number, it is possible to effectively avoid misjudgment of target loss due to accidental matching errors. This method can comprehensively consider the performance of the target in multiple frames of data, ensure that the target is lost only after multiple consecutive failed matches, and avoid the influence of single errors. Setting an appropriate trajectory frame number threshold can improve stability and robustness, reduce the probability of misjudgment, and ensure reliable tracking of targets in complex environments.

[0197] See also Figure 5 As shown, it is a flowchart of a video and radar acquisition target trajectory state provided by an embodiment of the present application. The whole process starts from S50 and is divided into two main process paths, which respectively process video and radar target detection.

[0198] S50: Start.

[0199] S5011: Obtain historical video detection targets.

[0200] First, the target information of the historical frames is obtained from the video source and preliminarily processed.

[0201] S5012: Video target state initialization.

[0202] Initialize the target states in the video data to ensure that each target can be correctly tracked from the start frame.

[0203] S5013: Construct parameter equations to estimate u and v respectively.

[0204] The trajectory parameter equation is established for the target's position information, and the horizontal and vertical pixels of the target in the video are estimated respectively.

[0205] S5014: Obtain the detection target of the current frame video.

[0206] Collect the video data of the current frame and perform target detection.

[0207] S5015: Construct a video frame optimization function.

[0208] A trajectory optimization objective is constructed to ensure the accuracy of video target detection, and the target's trajectory parameter equation is optimized to accurately estimate the target's position information.

[0209] S5016: Optimize parameter equations through iteration.

[0210] Optimize the trajectory parameter equations according to the trajectory optimization objective.

[0211] S5017: Target matching.

[0212] By comparing the positions of the targets between the current frames, target matching is performed to ensure the continuity of the targets in each frame.

[0213] S5018: Update target status.

[0214] Update the target's motion state according to the matching results and obtain trajectory information.

[0215] At the same time, the radar detection process starts from S5021. Similar to the video target detection process, the radar target is obtained and initialized, parameter equations are constructed, and finally target matching and updating are performed, which will not be repeated here.

[0216] S509: Does the target's status disappear?

[0217] The judgment step is to check whether the target is lost. If the target state disappears, the track of the target is deleted. If the target state is still valid, the tracking continues.

[0218] Through the above process, accurate detection and tracking of targets can be achieved, ensuring that targets in video and radar data can be accurately matched in each frame, and the target trajectory is updated according to the matching results. In the target matching process, the performance of the target in multiple frames is used to reduce misjudgments caused by accidental errors or short-term occlusions, ensuring the accuracy and stability of calibration.

[0219] See also Figure 6 FIG. 1 is a schematic diagram of a target trajectory provided in an embodiment of the present application. Figure 6The target trajectories of three targets A, B, and C in multiple time frames are shown. These trajectories are obtained by the position points where the targets appear in different time frames. Figure 6 In the above figure, the trajectory of each target is composed of multiple points, which represent the position change of the target in multiple time frames. By connecting these position points in time sequence, the motion trajectory of the target can be obtained, which is convenient for further analysis of the target trajectory.

[0220] S23: After the first feature points selected from the first target trajectories of different targets are formed into a first feature point group, each first feature point in the first feature point group and the second feature point on the corresponding second target trajectory are formed into a feature point pair.

[0221] Exemplarily, the selection of each first feature point in the first feature point group may be random, or may be selected based on certain conditions. In an optional implementation, the selection may be made based on the change of the motion state, position, or trajectory of the target.

[0222] For example, for a target that moves quickly or changes direction significantly, key points in its trajectory can be selected as feature points. For example, when a vehicle turns or accelerates, inflection points or speed mutation points in the trajectory can be selected as feature points, which can better reflect the dynamic characteristics of the target.

[0223] For another example, you can select points of the target in a specific area as feature points. For example, in a traffic scene, you can select points where a vehicle enters or leaves an intersection as feature points. These points usually have a high amount of information and can provide stronger constraints for subsequent calibration calculations.

[0224] For another example, for targets with obvious trajectory changes, extreme points (such as the highest point, the lowest point) or turning points in their trajectories can be selected as feature points. For example, in the trajectory of a vehicle, the turning points of acceleration, deceleration, turning or lane change can be selected as feature points. These points can better describe the motion characteristics of the target.

[0225] In the embodiments of the present application, the selection of feature points does not have to be limited to a specific rule, and can be flexibly performed according to specific needs. For example, points at specific locations or specific motion trajectories can be selected, or points can be randomly selected from multiple target trajectories to ensure that the selected feature points have sufficient representativeness and distribution range for optimization calculations in subsequent calibration steps. Regardless of which method is chosen, the key is to ensure that the selected feature points can effectively construct the corresponding relationship between targets in subsequent processing and provide necessary constraints for calibration calculations.

[0226] In an optional embodiment, in order to improve the accuracy and reliability of data calibration, the concept of calibration area is introduced. The calibration area divides the specified scene into several small spatial areas according to the direction of data collection and the spatial layout of the scene. Each calibration area represents a part of the scene with similar geometric features and distance ranges. These areas can be along the direction of data collection (such as radar scanning direction). Specifically, the division of the calibration area is based on the geometric characteristics of the collected data and the target distribution characteristics. For example, in radar data, the distance information of the target will affect the range of the calibration area, and the motion characteristics and trajectory changes of close-range targets and long-range targets may be quite different. Therefore, these targets can be divided into different calibration areas. In video data, the camera's viewing angle will affect the target viewing angle in different areas, which will also affect the accuracy of target detection. Therefore, by dividing the viewing angle range, it is also helpful to calibrate the data more accurately.

[0227] See also Figure 7 FIG. 1 is a schematic diagram of dividing a calibration area provided in an embodiment of the present application. Figure 4 The radar image and video image are shown in Figure 1, which are divided into three calibration areas by frame 1, frame 2 and frame 3. Each calibration area participates in the calculation according to the motion trajectory of the target within it, and the corresponding feature point constraint relationship is constructed to finally obtain the calibration matrix.

[0228] In this process, the calibration area is not only related to the spatial distribution of the target, but also closely related to the motion pattern of the target in different areas. The target trajectory in each area will be processed according to specific geometric constraints and feature point matching, and the final calibration matrix will be optimized through the constraint relationship of these feature points. For example, box 1 represents the short-range target area, box 2 represents the medium-range target area, and box 3 represents the long-range target area. The target trajectory, motion state and mutual relationship in each area will affect the selection of feature points and the calculation of the calibration matrix.

[0229] It should be noted here that this application does not specifically limit the number of calibration areas. The specific number of divisions can be flexibly adjusted according to the needs of the actual application, the distribution of the collected data, and the complexity of the scene. If the number of targets is small or the scene is simple, fewer calibration areas can be selected for processing to simplify the calculation process; for complex dynamic scenes, more calibration areas may be required to improve the accuracy and reliability of data processing. The number of calibration areas and their division method will be optimized based on multiple factors such as the field of view of the acquisition equipment, data characteristics, target type and its spatial distribution, thereby effectively improving the accuracy of data calibration.

[0230] The division of the calibration area can also be adjusted in a variety of ways, such as according to the target's movement speed, detection accuracy requirements, or the area can be divided more finely to facilitate data processing and calibration for different target groups. Data requirements in different scenarios may lead to different division methods of the calibration area, thereby ensuring the accuracy and reliability of the calibration results.

[0231] In the embodiments of the present application, the purpose of introducing the calibration area is to reduce the influence of factors such as the geometric shape and distance difference of different areas in the scene on the calibration results. Specifically, in practical applications, each area in the scene may have different distance ranges, and the targets in different areas may have significant differences in spatial position and movement mode. If the area is not divided and the calculation is performed using unified calibration parameters, the accumulation of calibration errors will result, especially when the scene is at different distances and the ground is uneven, the error may be more significant. Therefore, by dividing the calibration area, calibration optimization can be performed for each area separately, so that the calibration parameters of each area can more accurately reflect the target characteristics in the area.

[0232] In each calibration area, the selected target feature points are usually key points that can represent the motion state and spatial distribution of the target in the area. These feature points must not only meet certain spatial position requirements, but also be able to be reliably matched in actual scenes through the multimodal features of radar and video data.

[0233] In an optional implementation, for the target trajectory in each calibration area, since there are a large number of points in the trajectory, if all points are used for calculation, the amount of calculation will increase significantly. In order to effectively extract feature points, a feature point extraction method of constructing a triangle is used.

[0234] Optionally, for each calibration area in the specified scene, perform the following operations:

[0235] The first feature points selected from the first target trajectories of different targets, located in the calibration area and capable of constructing a triangle, are formed into a first feature point group; wherein each calibration area is obtained by dividing the specified scene along the collection direction corresponding to the collection data;

[0236] In the embodiment of the present application, each calibration area is first divided, and the "first feature points" that can construct triangles within the area are selected from the trajectories of different targets. At this time, these feature points are trajectory points that can satisfy certain spatial relationships selected from each calibration area. In order to ensure the accuracy of the calculation, these points should be able to form valid triangles. The goal of constructing a triangle is to use the point relationship in space so that the selected points can provide meaningful geometric constraints in subsequent steps.

[0237] See also Figure 8 FIG. 1 is a schematic diagram of constructing a characteristic triangle provided in an embodiment of the present application. Figure 8 The method of selecting feature points and constructing valid triangles in each calibration area is described. In each calibration area, three first feature points that can form a valid triangle are selected from the trajectories of different targets. These feature points are not only located in the calibration area, but also meet certain geometric relationship and spatial distribution requirements to ensure that they can provide meaningful geometric constraints. The feature triangle of each calibration area is composed of three trajectory points. The selection basis of these trajectory points is that they can meet geometric constraints, such as the requirements of side length ratio and angle range, so as to form a valid triangle and ensure that their spatial relationship can provide effective constraints in subsequent steps. The vertices of all these feature triangles together constitute the first feature point group, which represents the motion state and spatial distribution of the target in the calibration area, and provides a reliable basis for the subsequent multimodal feature matching of radar and video data.

[0238] For each first feature point group, each first feature point in the first feature point group and a second feature point on the corresponding second target trajectory form a feature point pair.

[0239] In the embodiment of the present application, each selected first feature point will be matched with the corresponding target position in the video data to form a feature point pair. By matching the target position in the radar data with the target position in the video data, a spatial relationship between the radar data and the video data can be established, thereby ensuring the calibration of the radar and the video.

[0240] It should be noted here that what is introduced in the embodiment of the present application is to first select the first feature point from each first target track in the radar data, and then select the corresponding second feature point from the video data to form a feature point pair. It is also possible to first select the second feature point from the video data, and then pair it with the track point in the radar data to form a feature point pair. The present application does not specifically limit the order of selecting the pairing of feature point pairs, and can be adjusted according to the requirements of the actual scene, data and computing resource limitations to ensure the efficiency and accuracy of the calibration process. For example, in some scenarios, it may be preferred to select feature points from radar data, especially when the radar data provides a higher precision target location; in other scenarios, the video data may provide clearer visual information, and then you can choose to extract feature points from the video data first.

[0241] Through the above process, firstly, the error caused by the uneven distribution of targets in the scene during the calibration process can be effectively reduced, ensuring that the trajectory of the target in each area is processed independently and accurately. Secondly, by selecting feature points that are located in the calibration area and can construct a triangle, the spatial relationship constraints of the target position can be further improved, making the calibration result more stable and accurate. By pairing each first feature point with the second feature point on the corresponding second target trajectory, the spatial relationship between radar data and video data can be accurately established, ensuring that the joint calibration of radar and video data is more accurate. In addition, this calibration method can flexibly select feature points in radar data or video data for pairing, which improves the adaptability of the method, enables it to be optimized according to different scene requirements and data characteristics, improves the efficiency of calibration calculations, and can remove redundant data, improve calculation speed, and ensure the efficiency and accuracy of the entire calibration process.

[0242] In an optional implementation, in order to ensure calculation efficiency and avoid unnecessary calculation burden, it is necessary to screen the feature points.

[0243] Optionally, for each first feature point group, before each first feature point in the first feature point group and the second feature point on the corresponding second target trajectory are combined into a feature point pair, the first feature point may be determined in the following manner:

[0244] For each first feature point group, construct a feature triangle using the first feature points in the first feature point group;

[0245] According to the similarity between the feature triangles, each candidate triangle group is determined from the constructed feature triangles; each candidate triangle group includes: each feature triangle whose similarity meets a preset threshold;

[0246] For each candidate triangle group, only the first feature point group corresponding to one feature triangle in the candidate triangle group is retained.

[0247] In the embodiment of the present application, a feature triangle refers to a geometric shape formed by selecting three non-collinear trajectory points, which can effectively reflect the geometric relationship between the target position and motion. A candidate triangle group refers to a set of triangles that meet the similarity requirements among the multiple triangles screened. The preset threshold is a set similarity standard, and only triangles with a similarity higher than the threshold will be considered as valid candidate triangles.

[0248] In an optional implementation, the similarity can be determined by calculating geometric features of the triangles, such as angle difference and side length ratio. If the angle difference and side length ratio meet a preset similarity threshold, it indicates that the two triangles are more similar.

[0249] Through the above process, by screening out the feature triangle groups that meet the similarity requirements, redundant data can be effectively removed, and only those triangles with high geometric structure similarity can be retained, thereby reducing the amount of calculation and improving the calculation efficiency. By selecting feature triangles with high geometric similarity, it is helpful to establish more stable and accurate target position constraints and improve the accuracy of radar and video data calibration. At the same time, through the reasonable screening and sorting of feature points, the calculation process is made more efficient, avoiding excessive calculation of irrelevant or unstable feature points, and improving the overall performance of the calibration process.

[0250] See also Fig. 9 As shown, it is a flow chart of constructing a characteristic triangle provided by an embodiment of the present application. As shown in the figure, the process is as follows S91~S97:

[0251] S91: Start.

[0252] Specifically, in this step, the entire feature triangle construction process is started and necessary initialization operations are prepared for the subsequent steps.

[0253] S92: Initialize the trajectory.

[0254] The initialization phase mainly sets the necessary parameters and data structures to prepare for the subsequent steps. For example, the number of divisions (num) of the calibration area, the minimum and maximum number of feature points, the selected trajectory threshold, etc. It is also necessary to load the radar and video data of the current frame, or extract relevant trajectory data from the historical frames. At the same time, an empty feature point set and triangle set are initialized.

[0255] S93: Select a trajectory point within the calibration area.

[0256] In this step, according to the preset area division scheme, eligible trajectories are selected from each calibration area. The calibration area division is to apply different calibration parameters according to the data characteristics of different calibration areas. When selecting eligible trajectory points, the trajectory points selected in each area need to meet certain conditions, such as the distribution range of trajectory points, time continuity, number of trajectory points, etc.

[0257] S94: Points within the calibration area form a characteristic triangle.

[0258] The selected trajectory points will be used to construct triangle features (i.e., feature triangles, the same below). The core goal of constructing these triangle features is to extract global spatial relationships and reduce redundant data. The selected feature points need to meet the following conditions:

[0259] 1) The three points cannot be on the same trajectory at the same time;

[0260] 2) The three edges need to be arranged in descending order. The purpose of this ordering is to ensure that the geometric shape of each triangle is consistent during calculation, which helps to identify redundant triangles and improve calculation efficiency.

[0261] It is also necessary to record the trajectory numbers of the three vertices and the sequential numbers within the trajectory corresponding to the points. This information helps in the subsequent matching, optimization and redundancy removal steps.

[0262] S95: removing redundant feature triangles according to a threshold value.

[0263] Redundant triangle features are removed based on the numbering of triangles and the order of side lengths. Redundant feature triangles refer to triangles in the same area whose shapes or geometric features are too similar, resulting in information duplication, affecting calibration accuracy and computational efficiency. According to the numbering and side length sorting of triangles, if triangles with similar shapes or geometric features are found, they are considered redundant, and redundancy is removed by setting a similarity threshold, retaining only one of the redundant triangles. In this way, the computational burden can be reduced and those characteristic triangles with strong representativeness can be retained.

[0264] S96: Complete the construction of feature triangles.

[0265] At this point, the filtered and de-redundant feature triangles have been constructed, and the selected feature points can better contain the global information in the area.

[0266] S97: End.

[0267] This step indicates the completion of the entire feature triangle construction process. Next, the serial number of the corresponding point in the image can be found by the serial number of the point, and the position mapping relationship between the point pairs can be further constructed to prepare for the subsequent calibration process.

[0268] S24: Optimizing the calibration parameters to be solved according to the position mapping relationship between the first feature point and the second feature point in each feature point pair, and obtaining the calibration parameters between the radar data and the video data.

[0269] The position mapping relationship refers to the geometric relationship between the first feature point (e.g., the target point in the radar data) and the second feature point (e.g., the target point in the video data). It can be represented by a mathematical function or a transformation matrix, mapping the target position in the radar coordinate system to the pixel position in the video image, and vice versa. The establishment of this mapping relationship enables the radar and video data to correspond in the same space.

[0270] The calibration parameters to be solved refer to the parameters that need to be solved through optimization, which are usually variables used to describe the conversion relationship between the radar coordinate system and the video image coordinate system. These parameters may include internal and external parameters, such as the rotation matrix, translation vector, focal length, distortion coefficient, etc. from radar to camera. The purpose of calibration is to find these parameters so that the radar data and video data can be accurately matched and the spatial relationship between them can be described consistently.

[0271] Calibration parameters refer to the final parameters obtained through the calibration process, which usually include: the internal and external parameters of the camera (such as focal length, distortion coefficient, rotation matrix, translation vector, etc.) and the conversion parameters between radar and video images. The final calibration parameters can be solved through optimization to ensure the spatial consistency between radar data and video data.

[0272] In the embodiment of the present application, a mapping relationship between the first feature points is determined according to each feature point pair. The mapping relationship can be described by calibration parameters, mapping the target position in the radar coordinate system to the pixel position in the video image, and vice versa.

[0273] Then, by optimizing the calibration parameters to be solved, the calibration parameters are finally obtained, so that the radar data and video data can be accurately matched in space, thereby ensuring the consistency of the spatial relationship between the two.

[0274] Optionally, for each calibration area in the specified scene, perform the following operations:

[0275] A calibration optimization target is constructed according to the position mapping relationship between the first feature point and the second feature point in each feature point pair in the calibration area; wherein each calibration area is obtained by dividing the specified scene along the acquisition direction corresponding to the acquisition data; the calibration optimization target represents: the difference between the feature points predicted by the position mapping relationship and the actual feature points.

[0276] In an optional implementation, after the calibration area is divided, for each calibration area, a feature point pair in the calibration area is obtained, and a position mapping relationship of the feature point pair is constructed.

[0277] See also Fig.10 As shown, it is a flow chart of determining calibration parameters provided in an embodiment of the present application. As shown in the figure, the process is as follows S1001~S1007:

[0278] S1001: Start.

[0279] In this step, the entire target trajectory determination process is started and necessary initialization operations are prepared for the subsequent steps.

[0280] 1002: Initialize feature triangle.

[0281] In this step, the track numbers and vertex numbers of the feature triangles are initialized to provide necessary support for the subsequent steps. This initialization process includes operations such as feature point selection and calibration area division.

[0282] S1003: Select a characteristic triangle corresponding to the area.

[0283] In this step, the characteristic triangles in the calibration area are selected to ensure that the selected triangles can reflect the target motion state and spatial distribution in the area. At this time, representative characteristic triangles are selected from each calibration area, and these triangles are constructed by selecting trajectory points to form characteristic triangles that can provide effective geometric constraints.

[0284] S1004: Find the second feature point according to the vertex number of the feature triangle.

[0285] This step determines the second feature point in the video data by matching the vertex number of the feature triangle with the target position in the video data. This process constructs a feature point pair by matching the first feature point in the radar data with the second feature point in the video data. Through matching, the spatial mapping relationship between the radar data and the video data can be established, providing data support for subsequent calibration calculations.

[0286] S1005: Constructing a position mapping relationship between feature point pairs.

[0287] In this step, a homogeneous equation is constructed based on the data in the feature point pair through the following formula 22 to describe the position mapping relationship between the radar data and the video data. Specifically, the following formula 22 is used to construct an equation representing the spatial transformation relationship to ensure accurate transformation between the radar coordinate system and the video coordinate system.

[0288] Exemplarily, the embodiment of the present application is based on each first feature point in the data of the feature point pair And each second feature point ,in, Represents the feature point index, and constructs a homogeneous equation as shown in the following formula 22 to describe the position mapping relationship between the radar and the video:

[0289] Formula 22

[0290] Then, by further transforming the above formula 22, we can obtain the following formulas 23 to 25:

[0291] Formula 23

[0292] Formula 24

[0293] Formula 25

[0294] It should be noted here that the equal sign is still Instead of the normal equal sign, the H matrix can also be multiplied by any non-zero constant. In actual processing, h9 can be set to 1 (when it takes a non-zero value). Then according to the third line, remove this non-zero factor, and we have:

[0295] Formula 26

[0296] Formula 27

[0297] Arranged:

[0298] Formula 28

[0299] Formula 29

[0300] S1006: Solve the calibration parameters through iteration.

[0301] In this step, the position mapping relationship of all feature point pairs has been constructed through the above formulas 22 to 29. The conversion relationship between radar and video can be effectively described by these feature point pairs. Ideally, when there is no error or noise, the relationship between the radar coordinate system and the video coordinate system should match perfectly. However, in the actual calibration process, there are usually errors or noise, so further optimization is needed to reduce these errors.

[0302] Still taking the above formula 28 and formula 29 as an example, the constraint relationship that can be constructed for each feature point pair is:

[0303] Formula 30

[0304] Formula 30 is equivalent to Ah=0, which is the calibration result under the ideal calibration state. The ideal calibration state means that during the calibration process, if there is no error or noise, the relationship between the radar data and the video data is perfectly matched. In other words, under the ideal calibration state, the conversion relationship between the radar coordinate system and the video coordinate system is accurate. When there is error or noise in the calibration of radar data or video data, the actual calibration result will deviate from the ideal calibration state. At this time, the calibration parameters need to be adjusted through the optimization algorithm to minimize the error.

[0305] Construct calibration error:

[0306] Formula 31

[0307] In this process, the error function In it, A is the matrix calculated according to the geometric relationship between radar and video, and h is the calibration parameter to be optimized.

[0308] The goal of calibration optimization is to obtain the optimal calibration parameters by minimizing the error function, so that the error between radar data and video data is as small as possible. Based on the above calibration error, the calibration optimization objective is constructed as

[0309] Formula 32

[0310] Where j represents the number of feature point pairs. By minimizing the errors of all feature points, the calibration parameter with the smallest total error is the final result, ensuring the consistency of radar and video data in space.

[0311] Then, by continuously adding residual relationships, the minimum step size is set using the Gauss-Newton method for iterative optimization to obtain calibration parameters that minimize the calibration optimization target error. Specifically, the Gauss-Newton method iteratively adjusts the current calibration parameters to gradually reduce the error between the predicted feature points and the actual feature points until the error converges to a minimum. Each step in the optimization process calculates the current error and gradient, updates the calibration parameters, and finally obtains the most appropriate calibration result.

[0312] S1007: End.

[0313] This step ends the calibration process. Through the optimization algorithm in the previous step (such as the Gauss-Newton method), the calibration parameters can be gradually adjusted to minimize the error. Specifically, the difference between the feature points is calculated using the error function, and the calibration parameters are optimized by minimizing these differences. Finally, the obtained calibration parameters can accurately describe the mapping relationship between the radar data and the video data.

[0314] By performing optimization operations on each calibration area separately, the mapping relationship between radar and video data can be accurately adjusted to ensure their consistency in space. First, by constructing calibration optimization targets based on the feature point pairs in each calibration area and using the position mapping relationship to predict the differences between feature points, errors can be effectively reduced and calibration accuracy can be improved. Secondly, the optimization process minimizes these differences to ensure that the calibration parameters to be solved (such as rotation matrix, translation vector, etc.) can accurately describe the transformation relationship between radar and video coordinate systems. In this way, not only the matching degree of radar and video data is improved, but also the robustness and adaptability are enhanced, ensuring accurate calibration under different scenarios and environmental conditions. In addition, region-by-region optimization can reduce errors caused by local environmental changes, making the calibration process more accurate and efficient, and ultimately improving the overall performance in multi-sensor data fusion, target positioning, and environmental perception.

[0315] See also Fig.11 As shown, it is an overall flow chart of a data calibration method provided in an embodiment of the present application, and the specific steps are as follows S1101~S1108:

[0316] S1101: Start.

[0317] Start the overall process of the data calibration method and prepare to start the data calibration process. This step marks the beginning of the entire process and enters the calibration preparation stage.

[0318] S1102: Initialize position parameters.

[0319] First, the initial parameters required for the calibration process need to be set. This includes the initialization of position parameters (such as target position, initial trajectory, etc.) to ensure that subsequent calculations can be performed based on the correct initial values. This step helps to ensure the accuracy of various calculations throughout the calibration process and provides a basis for subsequent optimization.

[0320] S1103: Estimate trajectory parameters and optimize the trajectory.

[0321] In this step, the target trajectory is determined and optimized using the radar data of the historical frame and the position information of the video detection, combined with the radar and video data of the current frame. The use of historical frame information helps to improve the accuracy of the current trajectory and ensure that the movement of the target can be accurately described during the trajectory tracking process.

[0322] S1104: Divide the calibration area and select trajectory points within the calibration area.

[0323] According to the calibration area division parameter (for example, num), the calibration area is divided into several small areas (num calibration areas). In each small area, select appropriate trajectory points. These trajectory points are usually used for further calibration and optimization. The selection process must ensure that these points contribute the most to the calibration accuracy. The way of dividing the area will affect the quality and accuracy of the calibration results.

[0324] S1105: Construct characteristic triangles.

[0325] The feature triangle is constructed using the trajectory points selected in the calibration area. The construction of the feature triangle helps to remove redundant data and extract meaningful feature points through the geometric characteristics of the triangle. In this way, the amount of data can be effectively reduced while improving the efficiency of the calibration process. The feature triangle provides important support for the subsequent feature point extraction and optimization.

[0326] S1106: Construct feature point pairs using feature triangles.

[0327] After constructing the feature triangle, the corresponding image points are found using the feature point numbers and trajectory numbers. These image points will form a set of feature point pairs for the subsequent calibration process. The accurate construction of feature point pairs is the key to data calibration, as they directly affect the accuracy of the calibration parameter solution.

[0328] S1107: Solving calibration parameters.

[0329] Based on the optimized feature point pairs, the calibration parameters are solved. These calibration parameters are a crucial part of the data calibration process. By solving the calibration parameters, the accuracy of positioning and measurement can be ensured to ensure that the final calibration result can match the actual situation to the greatest extent.

[0330] S1108: End.

[0331] The end of the calibration process completes all steps of the data calibration method. In this stage, the calibration results are applied to the actual scene for subsequent target tracking, positioning and other tasks. The accuracy of the calibration directly affects the subsequent performance, so before this step, it is necessary to ensure that all calibration parameters have been accurately solved.

[0332] Through the data calibration method of the present application, the trajectory parameters are optimized through multiple steps, the calibration area is reasonably divided, and the characteristic triangle is constructed, so as to accurately extract the characteristic point pairs, and the calibration parameters are solved through these characteristic point pairs. Through this process, it is possible to ensure that the calibration has high precision and high efficiency, and achieve good results in practical applications. Each step complements each other to ensure the safety and accuracy of the calibration method.

[0333] In summary, in simple terms, the embodiment of the present application proposes a data calibration method without manual intervention. First, by combining the radar and video detection results of the historical frame and the current frame to estimate the target trajectory, more accurate target positioning and path tracking can be achieved, providing reliable data support for subsequent calibration optimization. Secondly, the scene is divided into multiple calibration areas by the set calibration area division parameters, and each calibration area selects the corresponding trajectory point, which helps to perform calibration optimization in different areas respectively, thereby avoiding the complexity in global optimization and improving the calibration accuracy. In addition, in each calibration area, by constructing triangular features and removing redundant points, key points are extracted, and the quality of feature points is effectively improved. Compared with the method in the related art, the calculation amount is smaller, and the point pair selection can consider the global relationship, while ensuring that the feature points used in the calibration calculation are more accurate and effective. The optimization of the Gauss-Newton method can accurately minimize the error, ensure that the calibration parameters between the radar and video data are optimized, and make the spatial relationship between the two more accurate. The method ensures the efficiency and operability of data calibration through clear steps and process design, avoids repeated calculations and waste of resources, and improves the overall execution efficiency.

[0334] Finally, through the method of calibration area division and feature point extraction, the scheme can adapt to changes in different scenarios and data, and perform optimization in different areas, enhancing the robustness and adaptability in complex environments.

[0335] In summary, the embodiments of the present application provide an efficient, accurate, and highly adaptable data calibration method that can accurately match radar and video data and improve the overall performance of the system in multi-sensor fusion such as autonomous driving and robot perception.

[0336] Based on the same inventive concept, the present application also provides a data calibration device. Fig.12 As shown, it is a schematic diagram of the composition structure of the data calibration device 1200, which may include:

[0337] The target detection unit 1201 is used to perform target detection on each frame of the collected data of the specified scene to obtain the position information of the target appearing in each frame of the specified scene; the collected data includes radar data and video data;

[0338] The trajectory determination unit 1202 is used to obtain, for each target, a first target trajectory of the target under the radar data and a second target trajectory of the target under the video data according to the error between the position information of the target in each historical frame and the estimated position information, and the position information of each target in the current frame;

[0339] The feature extraction unit 1203 is used to form a first feature point group from first feature points selected from first target trajectories of different targets, and then form a feature point pair from each first feature point in the first feature point group and a second feature point on a corresponding second target trajectory;

[0340] The calibration unit 1204 is used to optimize the calibration parameters to be solved according to the position mapping relationship between the first feature point and the second feature point in each feature point pair, so as to obtain the calibration parameters between the radar data and the video data.

[0341] Optionally, the calibration unit 1204 is specifically used for:

[0342] For each calibration area in the specified scene, perform the following operations:

[0343] According to the position mapping relationship between the first feature point and the second feature point in each feature point pair in the calibration area, a calibration optimization target is constructed; wherein each calibration area is obtained by dividing the specified scene along the acquisition direction corresponding to the acquisition data; the calibration optimization target represents: the difference between the feature points predicted by the position mapping relationship and the actual feature points;

[0344] Based on the calibration optimization objective, the calibration parameters to be solved are optimized to obtain the calibration parameters corresponding to the calibration area.

[0345] Optionally, the position information obtained by performing target detection based on the collected data is the actual position information; each estimated position information is: the position information of the trajectory point represented by the trajectory equation to be solved; the trajectory determination unit 1202 is specifically used for:

[0346] According to the error between the actual position information of the target in each historical frame and the estimated position information of the corresponding trajectory point, the trajectory optimization target is constructed;

[0347] Based on the trajectory optimization goal, after optimizing the trajectory equation to be solved, the estimated position information of the target in the current frame is predicted;

[0348] After target matching based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame, the target trajectory of the target under the collected data of each frame is obtained according to the target matching result and the actual position information of the target in each historical frame; the target trajectory includes: the first target trajectory of the target under the radar data, and the second target trajectory of the target under the video data.

[0349] Optionally, the position information includes multiple position parameters, each position parameter corresponds to a trajectory equation to be solved; the trajectory determination unit 1202 is specifically used to:

[0350] For each position parameter, the trajectory optimization target corresponding to the position parameter is determined based on the error between the actual position parameter of the target in each historical frame and the estimated position parameter of the corresponding trajectory point;

[0351] For each position parameter, based on the trajectory optimization objective, the corresponding trajectory equation to be solved is optimized to obtain the optimized trajectory equation of the position parameter;

[0352] Based on the optimized trajectory equations of various position parameters, the estimated position information of the target in the current frame is predicted.

[0353] Optionally, the trajectory determination unit 1202 is specifically configured to:

[0354] Based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame, the matching errors between the target and each target are determined respectively;

[0355] Determine, among the matching errors, a target matching error that belongs to a preset matching error range and has a minimum error value;

[0356] The actual position information used to determine the target matching error is used as the target matching result of the target.

[0357] Optionally, the trajectory determination unit 1202 is further configured to:

[0358] If each matching error does not fall within the preset matching error range, then analyze whether the target's trajectory frame number meets the threshold; the trajectory frame number is: the frame number of the target's acquisition data in each frame of acquisition data;

[0359] If the preset threshold is not met, the target is determined to be lost.

[0360] Optionally, the feature extraction unit 1203 is specifically used for:

[0361] For each calibration area in the specified scene, perform the following operations:

[0362] The first feature points selected from the first target trajectories of different targets, located in the calibration area and capable of constructing a triangle, are formed into a first feature point group; wherein each calibration area is obtained by dividing the specified scene along the collection direction corresponding to the collection data;

[0363] For each first feature point group, each first feature point in the first feature point group and a second feature point on the corresponding second target trajectory form a feature point pair.

[0364] Optionally, for each first feature point group, before each first feature point in the first feature point group and the second feature point on the corresponding second target trajectory are combined into a feature point pair, the feature extraction unit 1203 is further configured to:

[0365] For each first feature point group, construct a feature triangle using the first feature points in the first feature point group;

[0366] According to the similarity between the feature triangles, each candidate triangle group is determined from the constructed feature triangles; each candidate triangle group includes: each feature triangle whose similarity meets a preset threshold;

[0367] For each candidate triangle group, only the first feature point group corresponding to one feature triangle in the candidate triangle group is retained.

[0368] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0369] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0370] After introducing the data calibration method and device according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0371] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0372] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server. In this embodiment, the structure of the electronic device may be as follows: Fig.13 As shown, it includes a memory 1301 , a communication module 1303 and one or more processors 1302 .

[0373] The memory 1301 is used to store computer programs executed by the processor 1302. The memory 1301 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0374] The memory 1301 may be a volatile memory, such as a random-access memory (RAM); the memory 1301 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1301 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1301 may be a combination of the above memories.

[0375] The processor 1302 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1302 is configured to implement the above data calibration method when calling the computer program stored in the memory 1301 .

[0376] The communication module 1303 is used to communicate with terminal devices and other servers.

[0377] The specific connection medium between the memory 1301, the communication module 1303 and the processor 1302 is not limited in the embodiment of the present application. Fig.13 In the embodiment, the memory 1301 and the processor 1302 are connected via a bus 1304. The bus 1304 is connected to the processor 1302 via a bus 1304. Fig.13 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1304 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.13 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0378] The memory 1301 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the data calibration method of the embodiment of the present application. The processor 1302 is used to execute the above-mentioned data calibration method, such as Figure 2 shown.

[0379] In some possible implementations, various aspects of the data calibration method provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the data calibration method according to various exemplary implementations of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 2 Follow the steps shown in .

[0380] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0381] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0382] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0383] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0384] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user electronic device, partially on the user electronic device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or completely on the remote electronic device or server. In the case of involving a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (e.g., using an Internet service provider to connect through the Internet).

[0385] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0386] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0387] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0388] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0389] These computer program commands may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the commands stored in the computer-readable memory produce a manufactured product including a command device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0390] These computer program commands may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the commands executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0391] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0392] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data calibration method, characterized in that: The method comprises: Performing target detection on each frame of the collected data of the specified scene to obtain the position information of the target appearing in each frame of the specified scene; the collected data includes radar data and video data; For each target, according to the error between the position information of the target in each historical frame and the estimated position information, and the position information of each target in the current frame, respectively obtain a first target trajectory of the target under the radar data and a second target trajectory of the target under the video data; After the first feature points capable of constructing triangles selected from the first target trajectories of different targets are formed into a first feature point group, for each first feature point group, respectively executing: constructing a feature triangle from the first feature points in the first feature point group; determining each candidate triangle group from the constructed feature triangles according to the similarity between the feature triangles; each candidate triangle group includes: each feature triangle whose similarity meets a preset threshold; for each candidate triangle group, only the first feature point group corresponding to one feature triangle in the candidate triangle group is retained; each first feature point in the first feature point group and the second feature point on the corresponding second target trajectory form a feature point pair; According to the position mapping relationship between the first feature point and the second feature point in each feature point pair, the calibration parameters to be solved are optimized to obtain the calibration parameters between the radar data and the video data.

2. The method according to claim 1, characterized in that The optimizing the calibration parameters to be solved according to the position mapping relationship between the first feature point and the second feature point in each feature point pair to obtain the calibration parameters between the radar data and the video data includes: For each calibration area in the specified scene, perform the following operations respectively: Constructing a calibration optimization target according to a position mapping relationship between a first feature point and a second feature point in each feature point pair in the calibration area; wherein each calibration area is obtained by dividing the specified scene along a collection direction corresponding to the collection data; and the calibration optimization target represents: a difference between a feature point predicted by the position mapping relationship and a real feature point; Based on the calibration optimization target, the calibration parameters to be solved are optimized to obtain the calibration parameters corresponding to the calibration area.

3. The method according to claim 1, characterized in that The position information obtained by performing target detection based on the collected data is the actual position information; each of the estimated position information is: the position information of the trajectory point represented by the trajectory equation to be solved; The method of obtaining a first target trajectory of the target under radar data and a second target trajectory of the target under video data respectively according to the error between the position information of the target in each historical frame and the estimated position information, and the position information of the target in the current frame, comprises: Constructing a trajectory optimization target according to the error between the actual position information of the target in each historical frame and the estimated position information of the corresponding trajectory point; Based on the trajectory optimization target, after optimizing the trajectory equation to be solved, the estimated position information of the target in the current frame is predicted; After target matching is performed based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame, the target trajectory of the target under the collected data of each frame is obtained according to the target matching result and the actual position information of the target in each historical frame; the target trajectory includes: a first target trajectory of the target under radar data, and a second target trajectory of the target under video data.

4. The method according to claim 3, characterized in that The position information includes a plurality of position parameters, each position parameter corresponds to a trajectory equation to be solved; the trajectory optimization target is constructed according to the error between the actual position information of the target in each historical frame and the estimated position information of the corresponding trajectory point, including: For each position parameter, determining a trajectory optimization target corresponding to the position parameter based on an error between the actual position parameter of the target in each historical frame and the estimated position parameter of the corresponding trajectory point; The step of optimizing the trajectory equation to be solved based on the trajectory optimization target and predicting the estimated position information of the target in the current frame includes: For each position parameter, based on the trajectory optimization target, the corresponding trajectory equation to be solved is optimized to obtain the optimized trajectory equation of the position parameter; Based on the optimized trajectory equations of the position parameters, the estimated position information of the target in the current frame is predicted.

5. The method according to claim 3, characterized in that The target matching based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame includes: Based on the estimated position information of the target in the current frame and the actual position information of each target in the current frame, respectively determining the matching error between the target and each target; Determine, among the matching errors, a target matching error that belongs to a preset matching error range and has a minimum error value; The actual position information used to determine the target matching error is used as the target matching result of the target.

6. The method according to claim 5, characterized in that The method further comprises: If the matching errors do not fall within the preset matching error range, then analyzing whether the target trajectory frame number meets the threshold; the trajectory frame number is: the frame number of the target acquisition data in each frame of acquisition data; If the preset threshold is not met, it is determined that the target is lost.

7. The method according to claim 1, characterized in that The first feature points capable of constructing a triangle selected from the first target trajectories of different targets are formed into a first feature point group, including: For each calibration area in the specified scene, perform the following operations respectively: The first feature points selected from the first target trajectories of different targets, located in the calibration area and capable of constructing a triangle, form a first feature point group; wherein each calibration area is obtained by dividing the designated scene along the collection direction corresponding to the collection data.

8. A data calibration device, characterized in that: include: The target detection unit is used to perform target detection on each frame of the acquired data of the specified scene to obtain the position information of the target appearing in each frame of the specified scene; The collected data includes radar data and video data; A trajectory determination unit is used to obtain, for each target, a first target trajectory of the target under radar data and a second target trajectory of the target under video data according to an error between the position information of the target in each historical frame and the estimated position information, and the position information of each target in a current frame; A feature extraction unit is used for selecting first feature points capable of constructing a triangle from first target trajectories of different targets to form a first feature point group, and then for each first feature point group, respectively performing the following steps: constructing a feature triangle from the first feature points in the first feature point group; According to the similarity between the feature triangles, determine each candidate triangle group from the constructed feature triangles; each candidate triangle group includes: each feature triangle whose similarity meets a preset threshold; for each candidate triangle group, only retain the first feature point group corresponding to one feature triangle in the candidate triangle group; each first feature point in the first feature point group and the second feature point on the corresponding second target trajectory form a feature point pair; The calibration unit is used to optimize the calibration parameters to be solved according to the position mapping relationship between the first feature point and the second feature point in each feature point pair, so as to obtain the calibration parameters between the radar data and the video data.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It includes a computer program, and when the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any method described in claims 1 to 7.

Images