Matching method and device, position acquisition method and system, equipment and medium

Abstract

The invention relates to the technical field of image processing, and provides a matching method and device, a position acquisition method and system, equipment and a medium, and the matching method comprises the following steps: S31, generating a space ray pointing to a target by taking the optical center of a camera as an original point; s32, constructing voxels of the laser radar point cloud; s33, searching voxels intersected with the space rays; and S34, taking the central point of the voxel as the position of the target in the space. According to the scheme, missing detection and false detection during fusion of the laser radar and the camera can be avoided, the calculation speed and precision are improved, and the position and track of the target are displayed in real time.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a matching method and apparatus, a location acquisition method and system, equipment and medium. Background Technology

[0002] With societal development, technologies such as intelligent driving, machine vision, and photogrammetry are being increasingly widely applied across various industries. The types of sensors used for data collection are also diverse, including LiDAR, cameras, and millimeter-wave radar. Each of these sensors has its own advantages and disadvantages, and they complement each other. Therefore, efficiently fusing data from multiple sensors is crucial for improving the effectiveness of perception algorithms.

[0003] Cameras generate 2D data, which can capture the color, shape, and other characteristics of objects, and the data is dense. Therefore, it is easy to find the outline of a target in an image, but a single camera cannot determine the actual size and volume of the target. LiDAR, on the other hand, generates 3D data, which can accurately determine the distance to an object, but the data is sparse. LiDAR cannot perceive information such as the color of the target, nor can it use color information to segment the target.

[0004] The main methods for detecting the spatial position of a target include monocular measurement, binocular measurement, and lidar measurement.

[0005] Monocular measurement using a single camera cannot independently pinpoint the location of a spatial target; it can only determine that the target lies on a straight line in space. It requires other conditions, such as determining that the target lies on a plane, which can be solved using line-plane intersection. However, monocular measurement cannot independently and directly obtain the target's position in space. Binocular measurement using two cameras can directly calculate the position of aerial targets by calculating the common perpendicular of two rays in space. However, to achieve accurate 3D measurement of targets in space, a certain distance needs to be maintained between the binocular cameras; the farther the target is from the cameras, the greater the distance between the two cameras must be. Binocular measurement requires setting up two cameras at a certain distance, and after setup, the relationship between the two cameras needs to be calibrated, making setup and calibration quite complicated.

[0006] LiDAR camera data fusion is a core technology essential for many applications, including autonomous vehicle localization, environmental mapping, and dynamic object recognition. Camera and LiDAR data are complementary; therefore, fusing camera image data with LiDAR point cloud data involves first extracting features from the image, then matching the corresponding points in the LiDAR point cloud to obtain the target's position in three-dimensional space. This multimodal data fusion method offers strong anti-interference capabilities and high real-time performance.

[0007] A single lidar unit can determine the position of a target in three-dimensional space. However, lidar point clouds are relatively sparse and discrete. When a single target is small, it may not be able to be correctly separated from the point cloud and may be filtered out as noise. Therefore, lidar-based spatial target measurement is not easy to complete target identification and segmentation, and is prone to missed detections and false detections.

[0008] Therefore, there is a need to provide a matching method and device, a location acquisition method and system, equipment and medium to avoid missed detections and false detections when fusion of lidar and camera, improve calculation speed and accuracy, and realize real-time display of target position and trajectory.

[0009] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0010] The main purpose of this invention is to overcome the problems of missed detection and false detection when fusion of lidar and camera, and to provide a matching method and device, a position acquisition method and system, equipment and medium to avoid missed detection and false detection when fusion of lidar and camera, improve calculation speed and accuracy, and realize real-time display of target position and trajectory.

[0011] To achieve the above objectives, the first aspect of the present invention provides a method for matching camera images and LiDAR point clouds, comprising the following steps: S31: Generate a spatial ray pointing towards the target with the camera's optical center as the origin; S32: Constructing voxels for lidar point clouds; S33: Locate voxels that intersect with the space ray; S34: The center point of the voxel is taken as the position of the target in space.

[0012] According to an exemplary embodiment of the present invention, generating a spatial ray pointing towards the target with the camera's optical center as the origin includes: Calculate the coordinates of the camera's optical center in the world coordinate system; Calculate the coordinates of the target in the image; Calculate the coordinates of the target in the world coordinate system on the normalized plane based on the target's coordinates in the image; The spatial ray equation is calculated based on the coordinates of the camera's optical center and the target in the world coordinate system.

[0013] According to an exemplary embodiment of the present invention, the voxels used to construct the lidar point cloud include: Given the size of a voxel; Calculate the voxel position coordinates of all point clouds; If a point cloud does not have a corresponding voxel, then create a voxel to which the point cloud belongs. Calculate the center point of all voxels.

[0014] According to an exemplary embodiment of the present invention, the search for voxels intersecting with the space ray includes: Calculate the coordinates of all pairs of vertex angles for each voxel; Calculate whether the projection of the spatial ray on each axis intersects with the voxel based on the coordinates of the vertex angle; If an intersection exists, the space ray is considered to intersect with the voxel.

[0015] According to an exemplary embodiment of the present invention, calculating whether the projection of the spatial ray on each axis intersects with the voxel based on the coordinates of the opposite angle includes: The two parameter sets when the space ray intersects the three faces of the voxel are obtained from the coordinates of the vertex. Calculate the first parameter when the space ray enters the voxel and the second parameter when it exits the voxel based on two parameter sets; If the first parameter is less than or equal to the second parameter, it indicates that there is an intersection between the space ray and the voxel.

[0016] According to a second aspect of the present invention, the present invention provides a matching device for camera images and lidar point clouds, comprising: a spatial ray generation module, a voxel generation module, an intersection calculation module, and a position generation module; The space ray generation module is used to generate a space ray pointing towards the target with the camera's optical center as the origin; The voxel generation module is used to construct voxels for lidar point clouds; The intersection calculation module is used to find voxels that intersect with space rays; The position generation module is used to locate the target in space with the center point of the voxel as the target's position.

[0017] According to a third aspect of the present invention, the present invention provides a method for obtaining a target location, comprising: S1: Calibrate the parameters of the camera and lidar; S2: The camera and lidar simultaneously capture images of the target; S3: Obtain the target's position in space based on the matching method between the camera image and the lidar point cloud.

[0018] According to a fourth aspect of the present invention, the present invention provides a target location acquisition system, comprising: a calibration device, an imaging device, and a matching device for the camera image and the lidar point cloud; The calibration device is used to calibrate the parameters of the camera and lidar; The imaging device includes a camera and a lidar, which simultaneously capture images of the target. The camera image and lidar point cloud matching device is connected to the calibration device and the shooting device to calculate the target's position in space.

[0019] As a fifth aspect of the present invention, the present invention provides an electronic device comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for matching camera images and LiDAR point clouds.

[0020] As a sixth aspect of the present invention, the present invention provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method for matching camera images and lidar point clouds.

[0021] The advantages of this invention are: This solution performs 3D measurement of spatial targets by fusing image and point cloud data. The camera and LiDAR can be pre-calibrated, eliminating the need for on-site calibration. Furthermore, the camera and LiDAR are integrated, requiring only a single site for 3D measurement, making it convenient to operate. It also boasts strong anti-interference capabilities; even if the number of reflected point clouds of a target in the LiDAR is small, it will not be filtered out, preventing missed detections. The calculation speed is fast and accurate, enabling real-time display of the target's position and trajectory. Attached Figure Description

[0022] The above and other objects, features, and advantages of this application will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0023] Figure 1 The schematic diagram illustrates the structure of a system for acquiring the target location.

[0024] Figure 2 The schematic diagram illustrates the structure of a device for matching camera images and lidar point clouds.

[0025] Figure 3 The diagram illustrates the steps of a method for obtaining the target location.

[0026] Figure 4 The diagram illustrates the steps of a method for matching camera images and LiDAR point clouds.

[0027] Figure 5 A schematic diagram illustrating the intersection of a space ray and a voxel is shown.

[0028] Figure 6 The schematic diagram illustrates the trajectory of the drone in a two-dimensional image.

[0029] Figure 7 The schematic diagram illustrates the trajectory of the drone in three-dimensional space.

[0030] Figure 8 A schematic diagram of the electronic device is shown.

[0031] Figure 9 A schematic diagram of the structure of a computer medium is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0033] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0036] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0037] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing this application, and therefore cannot be used to limit the scope of protection of this application.

[0038] According to a first specific embodiment of the present invention, the present invention provides a target location acquisition system, such as... Figure 1 As shown, it includes: a calibration device, an imaging device, and a matching device for camera images and lidar point clouds.

[0039] The calibration device is used to calibrate the parameters of the camera and lidar.

[0040] The imaging device includes a camera and a lidar sensor, which simultaneously capture images of the target.

[0041] The camera image and lidar point cloud matching device is connected to the calibration device and the shooting device to calculate the target's position in space.

[0042] like Figure 2 As shown, the matching device between camera images and lidar point clouds includes: a spatial ray generation module, a voxel generation module, a intersection calculation module, and a position generation module.

[0043] The space ray generation module is used to generate a space ray pointing towards the target with the camera's optical center as the origin.

[0044] The voxel generation module is used to construct voxels for lidar point clouds.

[0045] The intersection calculation module is used to find voxels that intersect with space rays.

[0046] The position generation module is used to locate the target in space with the center point of the voxel as the target's position.

[0047] Multimodal fusion of LiDAR and cameras effectively combines the three-dimensional spatial geometric information (point cloud, echo intensity, etc.) provided by LiDAR with the two-dimensional semantic information (such as RGB images) provided by the camera through techniques such as spatial alignment, feature transformation, or attention mechanisms in a unified three-dimensional space (such as bird's-eye view / BEV space) or feature space. This leverages the complementary advantages of the two modalities—the precise spatial location information of LiDAR and the rich semantic information of the camera—to enhance the performance of tasks such as 3D target detection, tracking, or segmentation.

[0048] According to a second embodiment of the present invention, the present invention provides a method for obtaining a target location, employing the target location acquisition system of the first embodiment, such as... Figure 3 As shown, it includes: S1: Calibrate the parameters of the camera and lidar.

[0049] Camera intrinsic parameters can be calibrated using a checkerboard pattern. By fixing the camera and taking images of the calibration board at different positions and orientations, the camera's intrinsic parameters and distortion can be determined using the Zhang Zhengyou calibration method.

[0050] Extrinsic parameter calibration of the camera with LiDAR. First, the LiDAR acquires a point cloud over a certain period of time in a feature-rich static scene, with the integration time controlled to around 30 seconds. Then, the camera captures an image of the scene. At least six feature points are manually selected in the image, and the same number of corresponding feature points are manually selected in the point cloud. These feature points should be non-coplanar and as evenly distributed as possible in both the image and the point cloud. Based on the manually selected corresponding points, the PNP algorithm is used to calculate the camera's extrinsic parameters. As long as the relative relationship between the camera and the LiDAR remains unchanged, recalibration is not required in other scenes.

[0051] In addition to manually selecting corresponding points in the image and point cloud, a cross-modal feature matching algorithm can be used to find corresponding matching points in the image and point cloud, and then the PNP algorithm can be used to calculate the extrinsic parameters.

[0052] Once calibration is complete, points in three-dimensional space can be projected onto two-dimensional points in the image. The projection equation is as follows: ; Where K is the camera intrinsic parameter, R and T are part of the camera extrinsic parameters, R is the rotation matrix in the extrinsic parameters, T is the translation vector in the extrinsic parameters, and X... L Y L Z L Let x, y, and z be the x, y, and z coordinates of a point in a 3D point cloud. u and v are the x and y coordinates of the point cloud projected onto the image. This is the scaling factor.

[0053] For a three-dimensional point in space, the projection equation can be used to determine a two-dimensional point on an image. However, given a two-dimensional point on an image, the target position in space cannot be obtained; only a spatial direction can be obtained. Other information is needed to determine the spatial position.

[0054] S2: The camera and lidar simultaneously capture images of the target.

[0055] The target can be an aircraft, such as a drone.

[0056] The camera and lidar can be set up on the ground.

[0057] S3: Obtain the target's position in space based on the matching method between camera images and LiDAR point clouds.

[0058] like Figure 4 As shown, the matching method between camera images and LiDAR point clouds includes the following steps: S31: Generate a spatial ray pointing towards the target with the camera's optical center as the origin.

[0059] If a projection point of the target in the image is known, and the camera's intrinsic parameters K and extrinsic parameters R and T are also known, a ray in space can be found, and the target is located on this ray.

[0060] S311: Calculate the coordinates of the camera's optical center in the world coordinate system.

[0061] The camera's optical center has coordinates (0,0,0) in the camera coordinate system, and its corresponding coordinates p0 in the world coordinate system are: .

[0062] Where R is the rotation matrix in the extrinsic parameters, and T is the translation vector in the extrinsic parameters.

[0063] S312: Calculate the coordinates of the target in the image.

[0064] Steps S311 and S312 can be performed simultaneously.

[0065] For objects in images, deep learning methods, such as the YOLO method, can be used for object detection. YOLO (You Only Look Once) is an object detection algorithm whose core idea is to transform the object detection task into a regression problem. It uses a convolutional neural network to perform inference directly on the image, predicting bounding boxes and class probabilities directly from image pixels, thus achieving end-to-end real-time object detection.

[0066] There are many versions of the YOLO method. This approach uses the YOLOv8 model, released in 2023. It incorporates the attention mechanism from the Transformer architecture, enhancing the model's ability to capture global image features. This allows the model to better understand the relationship between the target and the background in the image, thereby improving detection accuracy. Furthermore, it employs advanced training techniques, such as progressive training, improving training efficiency and convergence speed. Through reasonable design and optimization, YOLOv8 achieves significant improvements in both detection accuracy and speed.

[0067] The target detection module can detect the location of the target in the image. Given a target bounding box in the image, the coordinates of the center point of the target bounding box can be regarded as the pixel coordinates of the target (the coordinates of the target in the image).

[0068] S313: Calculate the coordinates of the target in the world coordinate system based on the target's coordinates in the image.

[0069] Assuming the coordinates of the projection point (target) in the image are (u, v), its coordinates p1 in the world coordinate system are: ; Where K is the camera intrinsic parameter, R and T are part of the camera extrinsic parameters, R is the rotation matrix in the extrinsic parameters, T is the translation vector in the extrinsic parameters, and u, v are the pixel coordinates of the image point.

[0070] S314: Calculate the spatial ray equation based on the coordinates of the camera optical center and the target in the world coordinate system.

[0071] Given two points in space, a straight line in space can be determined. The formula for a straight line in space passing through a point is: ; in, nrom is the normalized vector between two points, k is a coefficient greater than 0, p0 is the coordinate of the optical center in the world coordinate system, and p is any point on the line. Given a k, the specific coordinates of point p on the space ray can be calculated.

[0072] S32: Construct voxels for lidar point clouds.

[0073] Theoretically, a straight line from the camera's optical center through an image point will hit a point in the point cloud, which represents the target's location in space. However, in reality, point clouds are sparse, and this ray may not hit any point; that is, no point in the cloud will fall exactly on this straight line. Therefore, this solution addresses this issue by constructing voxels. A voxel in the point cloud is a small cube, and the probability of a ray hitting a cube is much higher than hitting a point. To quickly check whether the voxel containing a specific point has been created, a hash table is used to store the voxels.

[0074] S321: Given the size of the voxel.

[0075] A voxel is short for volume element. A solid containing voxels can be represented by stereo rendering or by extracting polygonal isosurfaces from a given threshold contour.

[0076] In this scheme, the voxel is a cube, and the size of the voxel is the side length of the cube, denoted as s, which can generally be set to 0.5m or 1m. The smaller the target to be detected, the smaller the voxel should be.

[0077] S322: Calculate the position coordinates of all voxels in the point cloud.

[0078] Traverse all point clouds. For any point cloud coordinates, first calculate the voxelPos of its corresponding voxel, which is to divide the x, y, z coordinates of the point cloud by the voxel size and round down, as shown in the following formula: ; Among them, X L Y L Z L Let be the coordinates of the point cloud, and s be the size of the voxel. This indicates rounding down, and voxelPos indicates the position of the voxel in the point cloud.

[0079] S323: If a point cloud does not have a corresponding voxel, then create a voxel to which the point cloud belongs.

[0080] The system checks if the voxel at that location has already been created in the hash table. If it has not been created, a voxel is created in the hash table, and the point cloud points are stored in that voxel. If the voxel already exists, the point cloud points are added directly to the voxel.

[0081] S234: Calculate the center point of all voxels.

[0082] Once the point cloud has been traversed, all voxels are completely determined; every point in the cloud corresponds to a voxel. Finally, the center point (voxelCenter) of each voxel is calculated, which represents the coordinates of the cube's center. The formula is as follows: ; Where voxelPos is the position coordinate of the voxel, and s is the voxel size.

[0083] S33: Find voxels that intersect with the space ray.

[0084] S331: Calculate the coordinates of a pair of vertex angles for all voxels.

[0085] First, traverse all voxels and calculate the coordinates of the two opposite vertices of each voxel. In a preferred implementation, one of the opposite vertices is closest to the origin and the other is farthest from the origin. In reality, it is the point with the smallest and largest coordinates on the three axes.

[0086] For example, if the range of the cube is: x-axis [x0,x1], y-axis [y0,y1], z-axis [z0,z1], then for the 8 vertices of the cube, the two chosen are (x0,y0,z0) and (x1,y1,z1).

[0087] There are two main reasons for choosing these two points to represent this voxel: first, given these two points, the original voxel size and position can be reconstructed; second, these two points are highly representative and easy to calculate.

[0088] Taking the coordinates of the bottom left vertex cornerMin and the top right vertex cornerMax as an example: ; ; Where volelCenter is the coordinate of the voxel center point, and s is the voxel size.

[0089] S332: Calculate whether the projection of the spatial ray on each axis intersects with the voxel based on the coordinates of the vertical angle.

[0090] The Slab method is used to calculate whether a ray intersects a voxel. This involves calculating the projection range of the ray on each axis (x, y, z axes of the world coordinate system), determining whether these ranges intersect, and thus determining whether the ray intersects the voxel.

[0091] In steps S311, S313, and S314, the calculated rays are in the world coordinate system (radar coordinate system), and the voxels are also in the radar coordinate system (world coordinate system). Therefore, the intersection calculation is also performed in the world coordinate system.

[0092] S3321: Based on the coordinates of the vertical angles, obtain the two sets of parameters when the space ray intersects the three faces of the voxel.

[0093] The voxel cube has six faces, and t1 and t2 are the parameter sets when the space ray intersects with three faces of the voxel.

[0094] ; ; in, Let be the direction vector of the spatial ray. Let be the coordinates of the camera's optical center in the world coordinate system. The division in the formula above represents the division of the x, y, and z components.

[0095] S3322: Calculate the first parameter when the space ray enters the voxel and the second parameter when it exits the voxel based on two parameter sets.

[0096] Calculate the parameters tMin when the ray enters the voxel and tMax when it exits the voxel: ; .

[0097] S3323: If the first parameter is less than or equal to the second parameter, it means that there is an intersection between the space ray and the voxel.

[0098] If tMin <= tMax, it means that the ray and the voxel intersect; otherwise, they do not intersect.

[0099] S333: If there is an intersection, the space ray is considered to intersect with the voxel.

[0100] S34: The center point of the voxel is taken as the position of the target in space.

[0101] If a spatial ray intersects a voxel, the traversal search stops, and the center point of that voxel, volelCenter, is the location of the target in space.

[0102] According to a third embodiment of the present invention, the present invention provides a trajectory acquisition system for a target location, including a trajectory generation system and a target location acquisition system according to the first embodiment.

[0103] The trajectory generation system generates the trajectory of the target in space based on the multiple locations of the target in space within a time period obtained by the target location acquisition system.

[0104] According to a fourth specific embodiment of the present invention, the present invention provides a method for obtaining a target location trajectory, comprising the following steps: The target's position in space is obtained using the target position acquisition method of the second specific embodiment; Connecting multiple locations of a target in space within a time period generates the target's trajectory in space.

[0105] like Figure 5As shown, for each frame of image and point cloud, the intersection point of the ray where the target is located and the voxel of the point cloud is calculated, and the continuous trajectory of the target in space can be obtained. Figure 5 This diagram displays the position of a drone at different times and its spatial ray as it flies through the air. Figure 5 The numerous rays in the image are rays that pass through the object's image point at different times, and they converge at the camera's optical center. Figure 5 The numerous cubes in the upper center represent point cloud voxels that were hit by rays at different times, indicating the drone's position at different moments.

[0106] like Figure 6 As shown, Figure 6 To show the location of the drone in a 2D image at different times, detected using the YOLO algorithm. Figure 6 The small black circles in the image represent the drone's location at different times.

[0107] like Figure 7 As shown, Figure 7 The method described above calculates the 3D coordinates of the UAV in a 3D point cloud space. Figure 7 The black curve in the image represents the drone's trajectory.

[0108] Therefore, this solution proposes a multimodal data fusion method for point cloud and image. The method calculates the position of the spatial target by voxelizing the point cloud and then determining whether the spatial ray passing through the image point intersects with the voxel. The target is detected by combining image and point cloud data, and the spatial position and trajectory of the target are calculated.

[0109] Compared with existing technologies, this solution uses image and point cloud data fusion for 3D measurement of spatial targets. It allows for pre-calibration of the camera and LiDAR, eliminating the need for on-site calibration. Furthermore, the camera and LiDAR are integrated, requiring only a single site for 3D measurement, making it convenient to operate. It also boasts strong anti-interference capabilities; even if the number of reflected point clouds of a target in the LiDAR is small, it will not be filtered out, preventing missed detections. The calculation speed is fast and accurate, enabling real-time display of the target's position and trajectory.

[0110] According to a fifth specific embodiment of the present invention, the present invention provides an electronic device, such as... Figure 8 As shown, Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0111] The following reference Figure 8 To describe an electronic device 100 according to this embodiment of the present application. Figure 8 The electronic device 100 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0112] like Figure 8 As shown, the electronic device 100 is presented in the form of a general-purpose computing device. The components of the electronic device 100 may include, but are not limited to: at least one processing unit 110, at least one storage unit 120, a bus 130 connecting different system components (including storage unit 120 and processing unit 110), a display unit 140, etc.

[0113] The storage unit stores program code that can be executed by the processing unit 110, causing the processing unit 110 to perform the steps described in this specification according to various exemplary embodiments of this application. For example, the processing unit 110 can perform the steps of the camera image and LiDAR point cloud matching method shown in the second embodiment.

[0114] The storage unit 120 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1201 and / or a cache storage unit 1202, and may further include a read-only memory unit (ROM) 1203.

[0115] The storage unit 120 may also include a program / utility 1204 having a set (at least one) program module 1205, such program module 1205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of these examples or some combination of these may include an implementation of a network environment.

[0116] Bus 130 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0117] Electronic device 100 can also communicate with one or more external devices 100' (e.g., keyboard, pointing device, Bluetooth device, etc.), enabling users to communicate with devices that interact with electronic device 100, and / or any device that allows electronic device 100 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 150. Furthermore, electronic device 100 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 160. Network adapter 160 can communicate with other modules of electronic device 100 via bus 130. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0118] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware.

[0119] Therefore, according to a sixth specific embodiment of the present invention, the present invention provides a computer-readable medium. For example... Figure 9 As shown, the technical solution according to the embodiments of the present invention can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) or on a network, and includes several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the above-described method according to the embodiments of the present invention.

[0120] The software product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0121] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0122] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0123] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the functions of the second specific embodiment.

[0124] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0125] Through the description of the above embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of the present invention.

[0126] Exemplary embodiments of the present invention have been specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, arrangements, or implementations described herein; rather, the present invention is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method of matching camera images and lidar point clouds, the method comprising: The method comprises the following steps: S31: generating a space ray pointing to the target with the camera optical center as the origin; S32: constructing a voxel of the laser radar point cloud; S33: finding the voxel intersecting with the space ray; S34: taking the center point of the voxel as the position of the target in space.

2. The method of matching camera images and lidar point clouds of claim 1, wherein, The generating of the space ray pointing to the target with the camera optical center as the origin comprises: calculating the coordinates of the camera optical center in the world coordinate system; calculating the coordinates of the target in the image; calculating the coordinates of the target in the world coordinate system on the normalized plane according to the coordinates of the target in the image; calculating the space ray equation according to the coordinates of the camera optical center and the coordinates of the target in the world coordinate system.

3. The method of matching camera images and lidar point clouds of claim 1, wherein, The constructing of the voxel of the laser radar point cloud comprises: giving the size of the voxel; calculating the position coordinates of the voxel to which all the point clouds belong; if the point cloud has no voxel to which it belongs, establishing the voxel to which the point cloud belongs; calculating the center point of all the voxels.

4. The method of matching camera images and lidar point clouds of claim 1, wherein, The finding of the voxel intersecting with the space ray comprises: calculating the coordinates of a pair of opposite corners of all the voxels; calculating whether the projection of the space ray on each axis intersects with the voxel according to the coordinates of the opposite corners; if there is intersection, it is considered that the space ray intersects with the voxel.

5. The method of matching camera images and lidar point clouds of claim 4, wherein, The calculating of whether the projection of the space ray on each axis intersects with the voxel according to the coordinates of the opposite corners comprises: obtaining two parameter sets when the space ray intersects with the three faces of the voxel according to the coordinates of the opposite corners; calculating the first parameter when the space ray enters the voxel and the second parameter when the space ray exits the voxel according to the two parameter sets; if the first parameter is less than or equal to the second parameter, it is considered that the space ray intersects with the voxel.

6. A device for matching camera images and lidar point clouds, characterized in that It comprises: a space ray generating module, a voxel generating module, an intersection calculating module and a position generating module; the space ray generating module is used for generating a space ray pointing to the target with the camera optical center as the origin; the voxel generating module is used for constructing a voxel of the laser radar point cloud; the intersection calculating module is used for finding the voxel intersecting with the space ray; the position generating module is used for calculating the center point of the voxel as the position of the target in space.

7. A method of acquiring a target position, characterized by, The method comprises the following steps: S1: calibrating the parameters of the camera and the laser radar; S2: simultaneously shooting the target by the camera and the laser radar; S3: obtaining the position of the target in space according to the matching method of the camera image and the laser radar point cloud in any one of claims 1-5.

8. An acquisition system of a target position, characterized in that, It comprises: a calibration device, a shooting device and the matching device of the camera image and the laser radar point cloud in claim 6; the calibration device is used for calibrating the parameters of the camera and the laser radar; the shooting device comprises the camera and the laser radar, and the camera and the laser radar simultaneously shoot the target; the matching device of the camera image and the laser radar point cloud is connected with the calibration device and the shooting device, and is used for calculating the position of the target in space.

9. An electronic device, comprising: It comprises: one or more processors; a storage device used for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the matching method of the camera image and the laser radar point cloud in any one of claims 1-5.

10. A computer readable medium having stored thereon a computer program, characterized in that, The programs are executed by the processors to implement the matching method of the camera image and the laser radar point cloud in any one of claims 1-5.

Images