A method, device and storage medium for combined radar and vision positioning

Abstract

An embodiment of this specification provides a method, device, and storage medium for combined radar and vision positioning, which can be applied to the field of intelligent transportation technology. The method includes: respectively performing internal parameter calibration on the camera and the radar; the camera and the radar respectively correspond to a camera coordinate system and a radar coordinate system; determining control points based on a reference point set and radar information; the radar information includes radar speed information, radar depth information, and radar curl information collected by the radar; determining the association relationship between the control points and the camera coordinate system in the world coordinate system; determining the central fusion relationship between the camera coordinate system and the world coordinate system; and mapping the radar information to the image coordinate system collected by the camera by integrating the association relationship and the central fusion relationship. The above method overcomes the problem of uncertain propagation of control points, reduces the error in the positioning process, improves the effect of combined radar and vision positioning, and is beneficial to positioning surrounding vehicles in practical applications.

Description

Technical Field

[0001] The embodiments of this specification relate to the technical field of intelligent transportation, and particularly to a method, device, and storage medium for combined radar and vision positioning. Background Art

[0002] In the application scenarios of intelligent transportation, perceiving and positioning the targets around a vehicle is one of the key technologies. Currently, cameras and millimeter-wave radars are generally used for perception. Since it is difficult for a camera to measure the position and speed of a target, and a millimeter-wave radar cannot distinguish the target category, the perception effect is generally optimized by means of multi-sensor fusion.

[0003] However, in actual applications, since the cameras, millimeter-wave radars, and the positions of the objects to be detected cannot be in an absolutely ideal state, the images and radar information do not completely correspond. Directly fusing the image and radar information will result in the combined information lacking a certain degree of accuracy. For example, directly superimposing the radar information on the image will result in a poor correlation between the centers of the image and radar information, thus affecting the positioning effect. Therefore, there is an urgent need for a method to optimize the combined radar and vision positioning effect. Summary of the Invention

[0004] The purpose of the embodiments of this specification is to provide a method, device, and storage medium for combined radar and vision positioning to solve the problem of how to optimize the combined radar and vision positioning effect.

[0005] To solve the above technical problems, the embodiments of this specification propose a method for combined radar and vision positioning, including: respectively performing internal parameter calibration on the camera and the radar; the camera and the radar respectively correspond to a camera coordinate system and a radar coordinate system; determining control points based on a reference point set and radar information; the radar information includes the radar speed information, radar depth information, and radar curl information collected by the radar; determining the association relationship between the control points and the camera coordinate system in the world coordinate system; determining the central fusion relationship between the camera coordinate system and the world coordinate system; and mapping the radar information to the image coordinate system collected by the camera by integrating the association relationship and the central fusion relationship.

[0006] An embodiment of this specification also provides a combined radar and vision positioning device, including: an internal parameter calibration module for calibrating the internal parameters of the camera and the radar respectively; the camera and the radar respectively correspond to a camera coordinate system and a radar coordinate system; a control point determination module for determining control points based on a reference point set and radar information; the radar information includes radar speed information, radar depth information, and radar curl information collected by the radar; a correlation relationship determination module for determining the correlation relationship between the control points and the camera coordinate system in the world coordinate system; a central fusion relationship determination module for determining the central fusion relationship between the camera coordinate system and the world coordinate system; an information mapping module for mapping the radar information to the image coordinate system collected by the camera by synthesizing the correlation relationship and the central fusion relationship.

[0007] An embodiment of this specification also provides a computer storage medium, on which a computer program is stored, and the computer program implements the steps of the above combined radar and vision positioning method when executed.

[0008] As can be seen from the technical solutions provided by the embodiments of this specification above, when performing combined radar and vision positioning, the embodiments of this specification first calibrate the internal parameters of the camera and the radar respectively to ensure the correction of the own parameters of the camera and the radar. Then, in combination with the radar information, control points are determined so that the control points are associated with the radar information, and further the correlation relationships between the control points and the camera coordinate system and the world coordinate system are determined, ensuring the strong correlation between the control points for determining the correlation relationships and different coordinate systems, and introducing the correlation between the radar coordinate system and the coordinate system corresponding to the camera. Then, by determining the central fusion relationship between the camera coordinate system and the world coordinate system, and then combining the correlation relationship, the radar information is mapped to the image collected by the camera, so that the objects in the image also have information such as depth and speed, realizing combined radar and vision positioning. Through the above method, the problem of uncertain propagation of control points is overcome, the error in the positioning process is reduced, the effect of combined radar and vision positioning is improved, and it is beneficial to position surrounding vehicles in practical applications. Description of the Drawings

[0009] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in this specification. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0010] Figure 1 It is a flowchart of a combined radar and vision positioning method according to an embodiment of this specification;

[0011] Figure 2Schematic diagram of the conversion from the world global coordinate system to the camera coordinate system in an embodiment of this specification;

[0012] Figure 3 Schematic diagram of the plane mapping from the camera to the image coordinate system in an embodiment of this specification;

[0013] Figure 4 Schematic diagram of establishing connections for control points in the world coordinate system in an embodiment of this specification;

[0014] Figure 5 Module diagram of a radar-vision integrated positioning device in an embodiment of this specification. Detailed implementation manners

[0015] Next, the technical solutions in the embodiments of this specification will be clearly and completely described with reference to the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this specification.

[0016] To solve the above technical problems, an embodiment of this specification proposes a radar-vision integrated positioning method. Specifically, the radar-vision integrated positioning method can be implemented by a radar-vision integrated positioning device, and the radar-vision integrated positioning device can be applied to intelligent transportation vehicles to detect driving information of the intelligent transportation vehicles, such as the distribution status of surrounding vehicles. As Figure 1 shown, the radar-vision integrated positioning method includes the following specific implementation steps.

[0017] S110: Perform internal parameter calibration for the camera and the radar respectively; the camera and the radar respectively correspond to a camera coordinate system and a radar coordinate system.

[0018] Since the manufacturing of devices cannot reach an absolutely ideal state, there are certain differences between the data obtained when the camera captures images or the radar acquires sensing information and the real data. For example, different images captured by a single camera will also produce various forms of distortion, which is called image distortion. To minimize the errors caused by the devices as much as possible, it is necessary to perform internal parameter calibration for the camera and the radar.

[0019] When calibrating the camera's internal parameters, the BRIEF (Binary Robust Independent Elementary Features) algorithm in computer vision can be used to detect targets and extract information from the two-dimensional images captured by the camera and map them into three-dimensional space. It is necessary to open up the optical path from the target object to the camera and even inside the camera, and model the entire physical path of the light. In order to model the optical model of the camera and enable the target in the three-dimensional world to be projected onto the two-dimensional image plane, what needs to be done is to transfer and unify multiple coordinate systems.

[0020] When extracting from a two-dimensional image acquired by a camera using the BRIEF algorithm, the image may first be subjected to Gaussian filtering to reduce noise interference.

[0021] After that, a certain size neighborhood window is obtained with the feature point as the center, for example, a neighborhood window of SxS. A pair of points are randomly selected in the window, and binary values ​​are assigned by comparing the pixel sizes of the two points. Specifically, let p(x) and p(y) be the random point x=(u 1 ,v 1 ),y=(u 2 ,v 2 ), the value can be assigned according to the formula conduct.

[0022] Select N = 256 pairs of random points from the neighborhood window, and ensure that the selected point x i ,y i All satisfied Gaussian distribution, and the sampling criterion obeys the isotropic Gaussian distribution. Repeat the above assignment operation for these random points to form a binary code, which constitutes the description of the feature point.

[0023] The above method optimizes the speed of the algorithm to a great extent and ensures the accuracy of the obtained binary code.

[0024] After obtaining the image features through the above steps, the coordinate system conversion can be performed to ensure that all data can be fused in the same coordinate system. The forward center axis of the camera coordinate system coincides with the optical axis of the lens in the opposite direction. It passes through the center of the camera lens and the center of the image and is perpendicular to the image plane. After introducing the four key coordinate systems, the conversion relationship of each coordinate system can be modeled.

[0025] Specifically, the conversion relationships between the world coordinate system and the camera coordinate system, the camera coordinate system and the image coordinate system, and the image coordinate system and the pixel coordinate system may be determined in sequence.

[0026] The world global coordinate system and the camera coordinate system are actually two independent three-dimensional coordinate systems. According to geometric principles, any two three-dimensional coordinate systems can be integrated into the same coordinate system through two means: coordinate rotation and coordinate translation. The rotation transformation from the world global coordinate system to the camera coordinate system is expressed using the unit orthogonal matrix R, and the translation transformation of the coordinate system is expressed using the column vector t. The rotation matrix R and the translation vector t constitute the extrinsic parameter matrix of the camera and the world global coordinate system. The conversion from the world global coordinate system to the camera coordinate system is as Figure 2 shown. Specifically, the conversion relationship between the world coordinate system and the camera coordinate system is In the formula, x c , y c , z c are the coordinates in the camera coordinate system, x w , y w , z w are the coordinates in the world coordinate system, R is the rotation matrix, and t is the translation vector.

[0027] Regarding the conversion relationship from the camera coordinate system to the image coordinate system, assume that the lens of the camera has the same principle as the light ray and the pinhole imaging principle. The light rays of the object point pass through the pinhole and hit the image plane. The distance from the lens center to the image plane is the focal length f. At the same time, according to the optical principle, the essence of pinhole imaging is a projection transformation from three-dimensional to two-dimensional, that is, the light rays pass through the points on the image plane. Using Figure 3 , the plane is placed in the front for projection. Therefore, after projection, the position information of the target object becomes two-dimensional. Point P in the figure is the coordinate in the image coordinate system, and point M is the coordinate in the world coordinate system. Specifically, the conversion relationship between the camera coordinate system and the image coordinate system is In the formula, x p , y p are the coordinates in the image coordinate system, x c , y c , z c are the coordinates in the camera coordinate system, and f is the focal length of the camera.

[0028] Regarding the conversion relationship from the image coordinate system to the pixel coordinate system, both the image coordinate system and the pixel coordinate system are two-dimensional coordinate systems and are in the same plane. However, their coordinate origins are not the same. In actual camera imaging, the origin of the image coordinate system has a certain offset relative to the center of the image. Assume that the position of the object coordinate point in the world global coordinate system projected onto the image coordinate system is (x c , y c ), then the conversion relationship between the pixel coordinate system and the image coordinate system is In the formula, u and v are the coordinates in the pixel coordinate system, x p , y p are the coordinates in the image coordinate system, u0 , v 0 is the origin position of the image coordinate system, d x , d y are respectively the width and height of the pixel point corresponding to the image coordinate system.

[0029] Without considering the distortion of the camera, the result obtained according to the above formula is the internal parameter matrix Cipm of the camera, specifically

[0030] Through the above process, the conversion relationships of the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system are completed in sequence.

[0031] Correspondingly, the millimeter-wave radar also needs to be calibrated before use. In practical applications, the internal parameter calibration is generally carried out after the millimeter-wave radar is installed on the vehicle. The internal parameter calibration is mainly for calibrating the yaw angle, pitch angle, and roll angle of the radar. The pitch angle, roll angle, and yaw angle need to be calibrated using equipment such as a small-sized spirit level and a corner reflector. Ensure that the vehicle with the radar installed is parked on a horizontal ground. Determine the vehicle's traveling direction according to the vehicle body, determine the test area (a straight line) 5 - 20 m in front of the vehicle and perpendicular to the traveling direction, use a small-sized spirit level to calibrate the vertical and horizontal of the vehicle-mounted radar; adjust the corner reflector to the same height as the radar, select multiple points to place the corner reflector and read the angle values through the radar, and finally determine the pitch angle, roll angle, and yaw angle according to the straight-line slope and included angle fitted from the data of multiple points.

[0032] S120: Determine the control points based on the reference point set and radar information; the radar information includes the radar speed information, radar depth information, and radar curl information collected by the radar.

[0033] After completing the internal parameter calibration, the external parameter calibration of the radar-vision system is also required. The EPnP algorithm (End-to-End Probabilistic Perspective-n-Points) can be used for the external parameter calibration. The EPnP algorithm involves the selection of control points, and currently, the control points are randomly selected in practical applications. However, since the control points need to achieve the joint calibration between the coordinate systems corresponding to different sensors, and the randomly selected control points may not be applicable to different coordinate systems. For example, the control points in the pixel coordinate system may not be applicable to the radar information, resulting in poor correlation of the object center and a large number of overlapping depth values. And since the radar information needs to be enriched into the image in the subsequent steps, it is necessary to strengthen the correlation between different coordinate systems based on the control points.

[0034] Therefore, in the embodiments of this specification, control points are determined by combining radar information. Therefore, the initial control points can be determined based on the centroid of the reference point set and the radar information, and then other control points can be determined based on the initial control points.

[0035] Specifically, the formula is used to select the initial control points. In the formula, is the initial control point, n is the number of reference points, P i w is the reference point, f is the focal length of the camera, d x is the width of the pixel point in the image coordinate system, γ is the deflection angle of the radar curl information, X r , Y r is the projection representation coordinate of the radar depth information, t 1 , t 2 is the translation variable, is the centroid variable of the feature points in space under the world coordinate system.

[0036] S130: Determine the association relationship between the control points in the world coordinate system and the camera coordinate system.

[0037] After determining the control points in the above manner, the association relationship between the coordinate systems can be determined based on the control points. Specifically, the three-dimensional points in the world coordinate system can be represented as a linear combination of 4 control points and mapped in the camera coordinate system as: Correspondingly, the correspondence between the world coordinate system and the camera coordinate system can be determined based on the formula , where x w , y w , z w are the coordinates in the world coordinate system, γ is the radar depth information, θ is the radar curl information, f is the camera focal length, x c , y c , z c are the coordinates in the camera coordinate system.

[0038] Parameterize the image information and radar information parameters corresponding to the space object, perform the conversion mapping under the coordinate system, then select the control points under the coordinate system, and establish the connection between the control points and other feature points under the coordinate system. As Figure 4 shown, establish the relationship between the control points in the camera coordinate system and the world coordinate system, that is, transfer the control points together with other feature points to the camera coordinate system. After solving the coordinates of the control points in the camera coordinate system, and then establishing the connection between the radar coordinate system and the image coordinate system, and solving the pose of the object to be measured under the camera and the radar with depth, angle, and curl information, the joint calibration of the radar-vision sensor can be performed.

[0039] S140: Determine the central fusion relationship between the camera coordinate system and the world coordinate system.

[0040] Central position fusion is the spatial fusion in radar-vision fusion. After the internal and external parameters of the two sensors are calibrated, it is necessary to unify their centers in the same frame to lay the foundation for subsequent joint positioning tasks. The central fusion problem is mapped to a mathematical problem as follows: The world global coordinate system and the camera coordinate system are actually two independent three-dimensional coordinate systems. According to geometric principles, any two three-dimensional coordinate systems can be integrated into the same coordinate system through two means: coordinate rotation and coordinate translation. External parameter calibration is actually a process of collecting image and millimeter-wave radar data, calculating the minimum projection loss through projection, and solving the external parameter matrix M (composed of the rotation matrix R and the translation vector t).

[0041] Specifically, after the radar data undergoes data conversion, the sensor coordinates relative to each radar point are output. Before data fusion, it is necessary to calibrate the space of the camera and the millimeter-wave radar. After calibration is completed, the obtained R and t are used to project all millimeter-wave radar points onto the image plane. For each target on the image, the radar projected point cloud falling within the detection frame is cut to obtain the radar region. Within the radar region of interest, each radar point is matched with the target detected in the image, so as to accurately match each image target with a millimeter-wave radar point and obtain its position, speed and other information.

[0042] There are multiple regions of interest within the frame of the same target. The positions of each region of interest on the associated image may have overlapping or interfering association problems, resulting in incorrect matching when the target is matched with the point cloud. Eventually, after target fusion positioning, there is a huge deviation between the true position and the positioning position. At the same time, due to the extremely sparse millimeter-wave radar point cloud, it is difficult to simply perform filtering processing from the data perspective. When using the density-based clustering algorithm to remove low-density points, since the radar points of a single target often appear independently, it is easy to filter out the point cloud of a single target, resulting in reasons such as missed positioning after fusion.

[0043] S150: Map the radar information to the image coordinate system collected by the camera based on the comprehensive association relationship and central fusion relationship.

[0044] After determining the association relationship and the central fusion relationship, these relationships can be integrated to map the radar information into the image collected by the camera.

[0045] Specifically, the process of mapping radar information may be to first detect a target object from the image collected by the camera, where the target object includes a target vehicle; then determine the object position of the target object in the camera coordinate system, and then convert the object position in the camera coordinate system into the target position in the radar coordinate system. Finally, extract the radar information at the target position and map it to the target object in the image.

[0046] When the data of the target object only includes the horizontal distance x and the vertical distance y, after projection, the data dimension will be reduced from 2D to 1D. That is to say, only the pixel abscissa u in the pixel coordinate system can be obtained after projection. According to this process, the projection conversion equation formula from the camera coordinate system to the millimeter-wave radar coordinate system can be obtained where u is the pixel abscissa of the target object in the pixel coordinate system, f is the focal length of the camera, d x is the width of the pixel point in the image coordinate system, γ is the yaw angle, X w is the horizontal distance of the target object in the radar coordinate system, Y w is the vertical distance of the target object in the radar coordinate system, t x is the horizontal translation vector, t y is the vertical translation vector, u 0 is the abscissa of the origin of the image coordinate system.

[0047] The method for estimating the target height and width is mainly based on the imaging model. Assuming the target height is h, the width is w, the distance between the target and the sensor is γ, the camera focal lengths are f respectively, and the pixel height of the target object on the image is |y 1 -y 2 |, and the pixel width is |x 1 -x 2 |. Construct a correlation matching model based on the target pixel width and height, and the coordinates of the radar information in the image coordinate system are obtained as:

[0048] The target radar-vision combined coordinate system shows the state of the object's motion and needs to consider the previous and subsequent moments. Let represent the change in the coordinate position of the target in the previous and subsequent moments. The target has a uniformly accelerated motion with an acceleration of a k between the sensors, and there is an error e k . Furthermore, the motion model of the radar mapping in the image coordinate system can be obtained as and where is the position coordinate of the target object at the later moment, A, B, and C are coefficients, a k is the acceleration of the target object, e k is the sensor error, For the H matrix of the L-BFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno) algorithm, only by iterating the L-BFGS algorithm can the optimal position motion estimation model be obtained each time. k For the and

[0049] matrices, only by iterating the L-BFGS algorithm can the optimal position motion estimation model be obtained each time.

[0049] The following summarizes the above process. First, feature extraction and coordinate system conversion are performed through the BRIEF algorithm to complete the calibration of the camera's internal parameters. At the same time, a spirit level and a corner reflector are used to complete the calibration of the millimeter-wave radar's internal parameters. Then, by integrating the image data and radar information, the external calibration of the sensors is performed. The EPnP algorithm is used for joint calibration, and the central positions of the two coordinate systems are located. Finally, the effect of mapping the radar information onto the image data is completed.

[0050] Through the introduction of the above embodiments, it can be seen that when the method performs joint radar-vision positioning, first, the internal parameters of the camera and the radar are respectively calibrated to ensure the correction of the camera and radar's own parameters. Then, the control points are determined by combining the radar information, so that the control points are associated with the radar information, and then the association relationships between the control points and the camera coordinate system and the world coordinate system are determined to ensure the strong association between the control points and different coordinate systems, and the association between the radar coordinate system and the corresponding coordinate system of the camera is introduced. Then, by determining the central fusion relationship between the camera coordinate system and the world coordinate system, and then combining the fusion relationship to map the radar information into the image collected by the camera, so that the objects in the image also have information such as depth and speed, realizing joint radar-vision positioning. Through the above method, the problem of uncertain control point propagation is overcome, the error in the positioning process is reduced, the effect of joint radar-vision positioning is improved, and it is beneficial to position surrounding vehicles in practical applications.

[0051] Based on the above joint radar-vision positioning method, an embodiment of this specification also proposes a joint radar-vision positioning device. As Figure 5 shown, the joint radar-vision positioning device may include the following specific modules.

[0052] The internal parameter calibration module 510 is used to respectively calibrate the internal parameters of the camera and the radar; the camera and the radar respectively correspond to a camera coordinate system and a radar coordinate system.

[0053] The control point determination module 520 is used to determine the control points based on the reference point set and the radar information; the radar information includes the radar speed information, radar depth information, and radar curl information collected by the radar.

[0054] The association relationship determination module 530 is used to determine the association relationship between the control points and the camera coordinate system in the world coordinate system.

[0055] The central fusion relationship determination module 540 is used to determine the central fusion relationship between the camera coordinate system and the world coordinate system.

[0056] The information mapping module 550 is used to map the radar information to the image coordinate system collected by the camera by integrating the association relationship and the central fusion relationship.

[0057] Based on the above radar-vision joint positioning method, an embodiment of this specification further provides a radar-vision joint positioning device. The radar-vision joint positioning device may include a memory and a processor.

[0058] In this embodiment, the memory may be implemented in any suitable manner. For example, the memory may be a read-only memory, a mechanical hard disk, a solid-state drive, or a USB flash drive, etc. The memory may be used to store computer program instructions.

[0059] In this embodiment, the processor may be implemented in any suitable manner. For example, the processor may take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc. The processor may execute the computer program instructions to implement Figure 1 the steps of the corresponding radar-vision joint positioning method.

[0060] This specification also provides an embodiment of a computer storage medium. The computer storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk drive (HDD), a memory card, etc. The computer storage medium stores computer program instructions. When the computer program is executed, it implements the computer program instructions in the corresponding embodiment of this specification Figure 1 corresponding thereto.

[0061] The radar-vision joint positioning method introduced in the above embodiments can be applied to the field of intelligent transportation technology, and can also be applied to other technical fields, and there is no limitation thereto.

[0062] Although the process flow described above includes multiple operations that occur in a specific order, it should be clearly understood that these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel (for example, using a parallel processor or a multi-threaded environment).

[0063] Although the process flows described above include multiple operations that occur in a specific order, it should be clearly understood that these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel (e.g., using a parallel processor or a multi-threaded environment).

[0064] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present specification. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce means for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0065] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0066] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0067] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0068] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash RAM. The memory is an example of computer-readable media.

[0069] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0070] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, the embodiments of this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0071] The embodiments of this specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The embodiments of this specification can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the description of the method embodiment. In the description of this specification, the description of reference terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of this specification. In this specification, the schematic expression of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0073] The above description is only for the embodiments of this application and does not limit this application. For those skilled in the art, various changes and modifications can be made to this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims

1. A combined radar and vision positioning method, characterized in that, it includes: Performing internal parameter calibration for the camera and the radar respectively; The camera and the radar respectively correspond to a camera coordinate system and a radar coordinate system; Determining control points based on a reference point set and radar information; the radar information includes radar speed information, radar depth information, and radar curl information collected by the radar; Determining the association relationship between the control points in the world coordinate system and the camera coordinate system; Determining the central fusion relationship between the camera coordinate system and the world coordinate system; Mapping the radar information to the image coordinate system collected by the camera by integrating the association relationship and the central fusion relationship; Determining the control points based on the reference point set and the radar information includes: selecting an initial control point according to the centroid of the reference point set; which includes: using the formula to select the initial control point, where in the formula is the initial control point, n is the number of reference points,[ is the reference point, f is the focal length of the camera, d x is the width of the pixel point in the image coordinate system, γ is the deflection angle of the radar curl information, X r , Y r is the projection representation coordinate of the radar depth information, t 1 , t 2 is the translation variable,[ is the centroid variable of the feature point in the space under the world coordinate system.[ 2. The method according to claim 1, characterized in that, The performing internal parameter calibration for the camera and the radar respectively includes: Extract the image feature encoding from the images captured by the camera using the BRIEF algorithm; change the random point selection method in the BRIEF algorithm as follows, where the random points x i and y i are subject to a Gaussian distribution, and the sampling criterion follows an isotropic Gaussian distribution; Successively determining the conversion relationships between the world coordinate system and the camera coordinate system, between the camera coordinate system and the image coordinate system, and between the image coordinate system and the pixel coordinate system.

3. The method according to claim 2, characterized in that, The conversion relationship between the world coordinate system and the camera coordinate system is where x c , y c , z c are the coordinates in the camera coordinate system, and x w , y w , z w are the coordinates in the world coordinate system, R is the rotation matrix, and t is the translation vector; The conversion relationship between the camera coordinate system and the image coordinate system is where x p , y p are the coordinates in the image coordinate system, and x c , y c , z c are the coordinates in the camera coordinate system, and f is the focal length of the camera; The conversion relationship between the pixel coordinate system and the image coordinate system is where u and v are the coordinates in the pixel coordinate system, and x p , y p are the coordinates in the image coordinate system, u 0 , v 0 is the origin position of the image coordinate system, and d x , d y are the width and height corresponding to the pixel point in the image coordinate system respectively.

4. The method according to claim 1, characterized in that, The determining the association relationship between the control points in the world coordinate system and the camera coordinate system includes: Based on the formula determine the correspondence between the world coordinate system and the camera coordinate system, where x w , y w , z w are the coordinates in the world coordinate system, γ is the radar depth information, θ is the radar curl information, f is the camera focal length, x c , y c , z c are the coordinates in the camera coordinate system.

5. The method according to claim 1, characterized in that, The mapping the radar information to the image coordinate system collected by the camera by integrating the association relationship and the central fusion relationship includes: Detecting a target object from the image collected by the camera; the target object includes a target vehicle; Determining the object position of the target object in the camera coordinate system; Converting the object position in the camera coordinate system into the target position in the radar coordinate system; Extracting the radar information at the target position and mapping it onto the target object in the image.

6. The method according to claim 5, characterized in that, The determining the object position of the target object in the camera coordinate system includes: Determining the lateral distance x and the longitudinal distance y of the target object in the camera coordinate system; Using the formula to determine the pixel abscissa of the target object in the pixel coordinate system, where u is the pixel abscissa of the target object in the pixel coordinate system, f is the focal length of the camera, d x is the width of the pixel point in the image coordinate system, γ is the yaw angle, X w is the lateral distance of the target object in the radar coordinate system, Y w is the longitudinal distance of the target object in the radar coordinate system, t x is the lateral translation vector, t y is the longitudinal translation vector, u 0 is the abscissa of the origin of the image coordinate system; The converting the object position in the camera coordinate system into the target position in the radar coordinate system includes: Determine that the abscissa corresponding to the target position is and the ordinate is where f is the camera focal length, h is the height of the target object, w is the width of the target object, |x 1 - x 2 | is the pixel width of the target object in the image, |y 1 - y 2 | is the pixel height of the target object in the image.

7. The method according to claim 5, characterized in that, The extracting the radar information at the target position and mapping it onto the target object in the image includes: Map the radar information to an image in combination with the motion state of the target object; wherein, determine the motion estimation position of the target object in combination with a motion model; the motion model includes and In the formula,[[]] is the position coordinate of the target object at the next moment, A, B, and C are coefficients, a k is the acceleration of the target object, e k is the sensor error,[[]] is the H of the L-BFGS algorithm k matrix.[[]] 8. A combined radar and vision positioning device, characterized in that, it includes: An internal parameter calibration module for performing internal parameter calibration for the camera and the radar respectively; The camera and the radar respectively correspond to a camera coordinate system and a radar coordinate system; A control point determination module for determining control points based on a reference point set and radar information; the radar information includes radar speed information, radar depth information, and radar curl information collected by the radar; An association relationship determination module for determining the association relationship between the control points in the world coordinate system and the camera coordinate system; A central fusion relationship determination module for determining the central fusion relationship between the camera coordinate system and the world coordinate system; An information mapping module for mapping the radar information to the image coordinate system collected by the camera by integrating the association relationship and the central fusion relationship; Determining the control points based on the reference point set and radar information includes: selecting an initial control point according to the centroid of the reference point set; which includes: using the formula to select the initial control point, where in the formula is the initial control point, n is the number of reference points is the reference point, f is the focal length of the camera, d x is the width of the pixel point in the image coordinate system, γ is the deflection angle of the radar curl information, X r and Y r are the projection representation coordinates of the radar depth information, t 1 and t 2 are translation variables is the centroid variable of the feature points in space in the world coordinate system 9. A computer storage medium, on which computer program instructions are stored, characterized in that, The computer program instructions, when executed, implement the steps of the combined radar and vision positioning method according to any one of claims 1-7.

Images