Data fusion method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose a data fusion method, device and equipment and a storage medium. The method comprises: obtaining an original image collected by a camera when a target object is irradiated by a laser point cloud; the original image comprises the target object; obtaining pixel point cloud coordinates according to positions of the laser point cloud in the original image; obtaining a conversion relationship between a radar coordinate system and an imaging coordinate system according to the pixel point cloud coordinates, the camera coordinate system and the radar coordinate system; and fusing the laser point cloud and the original image according to the conversion relationship. This method has the advantage that the laser point cloud does not need to be converted into a depth image before being fused with the original image, but the coordinate systems are unified by coordinate transformation, thereby reducing the difficulty of data fusion, reducing the time consumed in the data fusion process, and improving the efficiency of point cloud data and image data fusion.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data fusion method, apparatus, device, and storage medium. Background Technology

[0002] With the development of the times, people are pursuing more efficient and precise production methods. In fields such as construction and manufacturing, different sensors are often needed to collect data on target objects. For example, lidar is used to acquire point cloud data, and cameras are used to acquire images of target objects. To comprehensively grasp various information about the target object, it is necessary to further fuse different data.

[0003] In existing technologies, methods for fusing point cloud information and 2D image information mainly rely on manual calibration. This involves manually calibrating to obtain the conversion relationship between point cloud information and depth image information, and then converting the point cloud information into depth image information according to this relationship. The fusion of depth image information and 2D image information then achieves the goal of point cloud and image fusion. However, this method is highly dependent on manual calibration, and the data conversion and fusion process is complex, resulting in low data fusion efficiency. Summary of the Invention

[0004] This application provides a data fusion method, apparatus, device, and storage medium to improve the efficiency of fusion between laser point cloud and planar image data.

[0005] In a first aspect, embodiments of this application provide a data fusion method, including:

[0006] Acquire the original image captured by the camera when the target object is illuminated by a laser point cloud; wherein the original image includes the target object;

[0007] The pixel point cloud coordinates are obtained based on the position of the laser point cloud in the original image;

[0008] Based on the pixel point cloud coordinates, camera coordinate system, and radar coordinate system, the transformation relationship between the radar coordinate system and the imaging coordinate system is obtained;

[0009] Based on the transformation relationship, the laser point cloud and the original image are fused.

[0010] Secondly, embodiments of this application also provide a data fusion apparatus, including:

[0011] The raw image acquisition module is used to acquire the raw image captured by the camera when the target object is illuminated by a laser point cloud; wherein the raw image includes the target object;

[0012] The pixel coordinate determination module is used to obtain the pixel coordinates based on the position of the laser point cloud in the original image;

[0013] The transformation relationship determination module is used to obtain the transformation relationship between the radar coordinate system and the imaging coordinate system based on the pixel point cloud coordinates, camera coordinate system, and radar coordinate system.

[0014] The data fusion module is used to fuse the laser point cloud and the original image according to the transformation relationship.

[0015] Thirdly, embodiments of this application also provide an electronic device, including:

[0016] One or more processors;

[0017] Memory, used to store one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement a data fusion method as described in the first aspect of the present application.

[0019] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data fusion method as described in the first aspect of the present application.

[0020] The technical solution of this application embodiment obtains the transformation relationship between the radar coordinate system and the imaging coordinate system by transforming the laser point cloud in the imaging coordinate system, camera coordinate system, and radar coordinate system. Based on this transformation relationship, the laser point cloud and the original image can be directly fused. The advantage of this approach is that it eliminates the need to convert the laser point cloud into a depth image before fusing it with the original image. Instead, it unifies the reference coordinate system through coordinate transformation, thereby reducing the difficulty of data fusion, reducing the time spent in the data fusion process, and improving the efficiency of fusing point cloud data and image data. Attached Figure Description

[0021] Figure 1 This is a flowchart of the data fusion method provided in Embodiment 1 of this application;

[0022] Figure 2A This is a flowchart of the data fusion method provided in Embodiment 2 of this application;

[0023] Figure 2B This is a schematic diagram of the perspective relationship provided in Embodiment 2 of this application;

[0024] Figure 2C This is a schematic diagram of the coordinate system positional relationship provided in Embodiment 2 of this application;

[0025] Figure 3 This is a structural diagram of the data fusion apparatus provided in Embodiment 3 of this application;

[0026] Figure 4 This is a structural diagram of an electronic device provided in Embodiment 4 of this application. Detailed Implementation

[0027] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present application, not the entire structure.

[0028] Example 1

[0029] Figure 1 This is a flowchart of the data fusion method provided in Embodiment 1 of this application. This embodiment is applicable to the fusion of laser point clouds and planar images. The method can be executed by a data fusion device, which can be implemented in software and / or hardware and specifically configured in an electronic device.

[0030] refer to Figure 1 The data fusion method shown includes the following steps:

[0031] S110. Acquire the original image captured by the camera when the target object is illuminated by a laser point cloud; wherein the original image includes the target object.

[0032] The target object can be any object for which laser point cloud and image fusion are performed. The laser point cloud can be points emitted by the lidar onto the target object, and the camera point cloud coordinates of these points in the camera coordinate system can be determined based on their spatial position. Simultaneously, when the lidar projects laser light onto the target object, a camera captures an image containing the target object, which is the original image. It is understood that when laser light is projected onto the target object, creating a laser point cloud, the original image captured by the camera will contain this laser point cloud.

[0033] S120. Obtain the pixel point cloud coordinates based on the position of the laser point cloud in the original image.

[0034] The pixel coordinates are the position coordinates of the laser point cloud in the original image. The imaging coordinate system can be a coordinate system established based on the original image, with the coordinate unit being pixels. The center point of the original image can be used as the origin of the imaging coordinate system. Based on the position of the laser point cloud in the original image and its positional relationship with the set origin, the pixel coordinates of the laser point cloud in the original image are derived.

[0035] It is understandable that, since the imaging coordinate system is established based on the original image, it is a two-dimensional coordinate system. Pixel point cloud coordinates can be the coordinate data of the laser point cloud in the original image under the imaging coordinate system.

[0036] S130. Based on the pixel point cloud coordinates, camera coordinate system, and radar coordinate system, obtain the transformation relationship between the radar coordinate system and the imaging coordinate system.

[0037] The radar coordinate system is established based on the lidar. It should be noted that, in practice, the lidar and camera cannot be positioned at the same location, resulting in differences in the position and orientation of the radar and camera coordinate systems. Therefore, to fuse the lidar point cloud data with the original image data, it is necessary to unify their reference coordinate systems.

[0038] Therefore, based on the pixel point cloud coordinates, camera coordinate system, and camera point cloud coordinates obtained in the preceding steps, the mapping relationship between the imaging coordinate system and the camera coordinate system can be determined. Furthermore, based on the position and attitude relationship between the camera coordinate system and the radar coordinate system, the transformation relationship between the radar coordinate system and the imaging coordinate system can be derived. The position and attitude relationship between the camera coordinate system and the radar coordinate system can be determined manually. This coordinate transformation can be performed based on principles such as perspective transformation and / or rigid body kinematics.

[0039] S140. Based on the transformation relationship, fuse the laser point cloud and the original image.

[0040] Based on the transformation relationship between the radar coordinate system and the imaging coordinate system obtained in the preceding steps, the coordinates of the laser point cloud in the radar coordinate system can be transformed to the imaging coordinate system and fused with the original image. The fusion method used can be any existing point cloud image fusion algorithm, and this application embodiment does not limit this approach.

[0041] The technical solution of this application embodiment obtains the transformation relationship between the radar coordinate system and the imaging coordinate system by transforming the laser point cloud in the imaging coordinate system, camera coordinate system, and radar coordinate system. Based on this transformation relationship, the laser point cloud and the original image can be directly fused. The advantage of this approach is that it eliminates the need to convert the laser point cloud into a depth image before fusing it with the original image. Instead, it unifies the reference coordinate system through coordinate transformation, thereby reducing the difficulty of data fusion, reducing the time spent in the data fusion process, and improving the efficiency of fusing point cloud data and image data.

[0042] In one alternative implementation, before fusing the laser point cloud and the original image according to the transformation relationship, the method may further include: correcting the transformation relationship based on a vertical reference line located on the target object.

[0043] The vertical reference line can be used to correct the transformation relationship between the radar coordinate system and the imaging coordinate system. Preferably, a vertical reference line perpendicular to the horizontal plane can be selected. Because the original image captured by the camera has certain distortions, some errors occur when the camera laser coordinates are transformed to pixel laser coordinates, and there are also measurement errors in the positional relationship between the camera coordinate system and the radar coordinate system. Therefore, a vertical reference line actually existing on the target object is used to help correct the transformation relationship between the radar coordinate system and the imaging coordinate system. The correction method can be any existing correction method, and this application embodiment does not limit it.

[0044] Optionally, horizontal reference lines parallel to the horizontal plane can be selected for auxiliary correction, as can vertical and / or horizontal reference lines appearing on other reference objects in the original image.

[0045] The above implementation method corrects the transformation relationship between the radar coordinate system and the imaging coordinate system by selecting a vertical reference line on the target object, thereby reducing the error of data fusion and improving the accuracy of data fusion.

[0046] In one optional implementation, the step of correcting the transformation relationship based on the vertical reference line located on the target object may include: determining the average level coordinates of each reference point cloud in the vertical reference line in the imaging coordinate system; and correcting the transformation relationship based on the average level coordinates according to a preset straight line detection algorithm.

[0047] Each reference point cloud refers to at least two reference point clouds located on the vertical reference line (in practice, the more reference point clouds selected, the better the correction accuracy). The horizontal coordinates are calculated based on the horizontal coordinates of each reference point cloud in the imaging coordinate system. Then, a preset line detection algorithm is used to calculate the offset of the horizontal coordinates, thereby correcting this offset in the transformation relationship. The preset line detection algorithm can be any existing line detection method, such as the Hough algorithm; this embodiment does not limit its use.

[0048] In the above implementation, the transformation relationship is compensated and corrected based on the average level coordinates of each reference point cloud and the preset straight line detection algorithm, which reduces the interference of various errors on data fusion during the fusion process and further improves the accuracy of data fusion.

[0049] In one alternative implementation, the method may further include: determining a binary image of the target object based on a transformation relationship; obtaining a tri-color image based on point cloud depth information and the binary image; and extracting a vertical reference line from the tri-color image.

[0050] Among them, binary images are black and white images, and tri-color images are RGB images (Red, Green, Blue). Point cloud depth information can be the depth information of the laser point cloud from the camera or the depth information of the laser radar.

[0051] Specifically, by utilizing the transformation relationship between the radar coordinate system and the imaging coordinate system, the coordinates of the laser point cloud (three-dimensional coordinates) are transformed to the imaging coordinate system (two-dimensional coordinates). This results in a mapping from the three-dimensional laser point cloud to a two-dimensional image, forming a binary image. Color information is then added to the binary image using the depth information of the laser point cloud. For example, different colors can be assigned to each pixel in the binary image based on its different depth information. Finally, an RGB image is obtained, and vertical reference lines and reference point clouds are selected from the RGB image. The selection of the vertical reference lines can be achieved using any method in the prior art, such as indirect methods; this application does not limit this approach.

[0052] Optionally, since the pixels in an RGB image are very small, it is not conducive to the selection of vertical reference lines and reference point clouds. Therefore, pixel augmentation can be used to assign the same color to the pixels surrounding the reference point cloud, thereby increasing the number of pixels. The pixel augmentation method can employ any existing image filtering method, and this application embodiment does not limit it.

[0053] The above implementation method assigns color to the transformed binary image based on the depth information of the laser point cloud, which improves the selection efficiency of the vertical reference line and thus helps to improve the correction efficiency in the data fusion process.

[0054] Example 2

[0055] Figure 2A This is a flowchart illustrating the data fusion method provided in Embodiment 2 of this application. This embodiment refines the determination of the transformation relationship based on the technical solutions of the foregoing embodiments, thereby improving the efficiency of fusion between laser point cloud and planar image data.

[0056] refer to Figure 2A The data fusion method shown includes the following steps:

[0057] S210. Acquire the original image captured by the camera when the target object is illuminated by a laser point cloud; wherein the original image includes the target object.

[0058] S220. Obtain the pixel point cloud coordinates based on the position of the laser point cloud in the original image.

[0059] Specifically, based on the position of the corresponding pixel in the original image, an imaging coordinate system is established based on the original image, and the pixel coordinates of the laser point cloud in the imaging coordinate system are determined with pixels as the unit length.

[0060] S230. Determine the mapping relationship between the imaging coordinate system and the camera coordinate system based on the pixel point cloud coordinates and the camera coordinate system.

[0061] Specifically, based on the spatial relationship between pixel point cloud coordinates and camera coordinate system, the transformation path between the imaging coordinate system and camera coordinate system can be obtained, which is called the mapping relationship.

[0062] In one optional implementation, determining the mapping relationship between the imaging coordinate system and the camera coordinate system based on the pixel point cloud coordinates and the camera coordinate system may include: determining the mapping relationship between the imaging coordinate system and the camera coordinate system based on the perspective relationship between the pixel point cloud coordinates and the laser point cloud in the camera coordinate system. Since the camera's shooting position is relatively fixed relative to the scene being photographed, the image plane simulated based on the camera's frame and the scene being photographed have a perspective relationship. Therefore, based on this perspective relationship, and according to the perspective transformation between the laser point cloud coordinates in the camera coordinate system and the pixel point cloud coordinates, the mapping relationship between the imaging coordinate system and the camera coordinate system can be derived, which can be expressed as a homogeneous coordinate equation or a homogeneous transformation matrix.

[0063] The technical solution described above, based on the perspective relationship between the image and the actual scene, can obtain the mapping relationship between the camera coordinate system and the imaging coordinate system without the need for complex algorithms, reducing a large amount of computation, saving time and computing resources for data fusion, and improving the efficiency of data fusion.

[0064] S240. Determine the rotation and translation relationship between the camera coordinate system and the radar coordinate system based on the pose of the camera coordinate system and the radar coordinate system.

[0065] The distance between the camera coordinate system and the radar coordinate system can be obtained through measurement; the distance between the origins of the two coordinate systems is the distance between their positions. The poses of the camera and radar coordinate systems can be determined during coordinate system establishment based on actual conditions. Since Cartesian coordinate systems are preferred for both the camera and radar coordinate systems in three-dimensional space, the rotation and translation relationship between them can be determined based on the kinematic transformations of a rigid body in Cartesian space. That is, different Cartesian coordinate systems in Cartesian space can coincide through rotation and translation. Therefore, the rotation and translation relationship between the camera and radar coordinate systems can be determined. This relationship can be expressed using homogeneous coordinate equations or homogeneous transformation matrices.

[0066] In one optional implementation, determining the rotation and translation relationship between the camera coordinate system and the radar coordinate system based on the pose of the camera coordinate system and the pose of the radar coordinate system may include: determining the rotation and translation relationship between the camera coordinate system and the radar coordinate system based on the distance components and attitude vectors between the two origins of the camera coordinate system and the radar coordinate system.

[0067] Specifically, based on the distance components between the origins of the camera coordinate system and the radar coordinate system, the translation process required for the two origins to coincide is derived. Furthermore, based on the attitude vector between the two coordinate systems (i.e., the unit vector of the coordinate system's direction axis), the rotation process required for the two coordinate axes to coincide is derived. These translation and rotation processes are then expressed using homogeneous coordinate equations or homogeneous transformation matrices, serving as the rotation-translation relationship between the camera coordinate system and the radar coordinate system.

[0068] S250. Based on the mapping relationship and the rotation and translation relationship, determine the transformation relationship between the radar coordinate system and the imaging coordinate system.

[0069] Based on the rotation and translation relationships between the radar coordinate system and the camera coordinate system, and the mapping relationship between the camera coordinate system and the imaging coordinate system obtained from the preceding steps, the transformation relationship between the radar coordinate system and the imaging coordinate system can be derived. Specifically, since both the rotation and translation relationships and the mapping relationship can be expressed in the form of homogeneous coordinate equations or homogeneous transformation matrices, the homogeneous transformation matrix used to represent the transformation relationship between the radar coordinate system and the imaging coordinate system can be obtained through the derivation of the matrix and the equation.

[0070] S260. Based on the transformation relationship, the laser point cloud and the original image are fused.

[0071] The technical solution of this application derives the transformation relationship between the radar coordinate system and the imaging coordinate system based on the rotation and translation relationship between the radar coordinate system and the camera coordinate system, and the mapping relationship between the camera coordinate system and the imaging coordinate system. This approach solves the problem of low data fusion efficiency caused by the need to convert laser point clouds into depth images in existing technologies. By unifying the reference coordinate system through coordinate transformation, the difficulty of data fusion is reduced, the time spent in the data fusion process is decreased, and the efficiency of fusing point cloud data and image data is improved.

[0072] Based on the above embodiments, this application also provides a preferred embodiment. Figure 2B This is a schematic diagram of the perspective relationship provided in Embodiment 2 of this application. For example... Figure 2B The diagram shows the perspective relationship, where:

[0073] Map the 3D point cloud P on the target object to a 2D image and determine the mapping relationship between P and the imaging coordinate system. Figure 2B In the middle, xc -y c -z c This represents the camera coordinate system, with its origin at O. c , z c The positive direction is the optical axis direction. Assume that the measured coordinates of P in the camera coordinate system are (x, y, z), and uv is the imaging coordinate system. Let the coordinates of P(x, y, z) in the imaging coordinate system be (u, v).

[0074] P(x,y,z) has a perspective relationship with the imaging coordinate system uv. According to the principle of similar triangles, the following relationship can be obtained:

[0075]

[0076]

[0077] Where f is the camera focal length, in pixels, and its length is O. c Distance to the two-dimensional image plane, (c x ,c y Let be the midpoint of the image. Then, we have homogeneous coordinates:

[0078]

[0079] Figure 2C This is a schematic diagram of the coordinate system positional relationship provided in Embodiment 2 of this application, as shown below. Figure 2C As shown: x c -y c -z c Denotes the camera coordinate system, x r -y r -z r This represents the radar coordinate system, with the origins of the two coordinate systems being O and O'. c and O r The distance components between the two origins are d x d y and d z ,according to Figure 2C The coordinate transformation relationship between the two coordinate systems can be obtained:

[0080] x c =-y r +d y

[0081] y C =-z r -d z

[0082] z C =x r -d x

[0083] Rewritten in homogeneous coordinate form, we get:

[0084]

[0085] In summary, the laser point cloud (x) can be obtained r ,y r ,z r The mapping relationship between the radar coordinate system and the image pixel (u,v), that is, the mapping equation between the radar coordinate system and the imaging coordinate system:

[0086]

[0087] Among them, Γ r_c This is the mapping matrix corresponding to the transformation relationship between the radar coordinate system and the imaging coordinate system.

[0088] The above-mentioned mapping matrix, based on perspective, corresponds to the transformation relationship between the radar coordinate system and the imaging coordinate system. However, in practical applications, there are certain errors in the position measurement between the camera and the lidar. Therefore, the mapping matrix Γ... r_c Further calibration is needed. Since lidar uses arc-shaped scanning lines, edge detection is easier; therefore, objects with longer vertical edges were chosen as calibration references.

[0089] The mapping from 3D laser point clouds to 2D images has been completed. In the obtained binary image, the point cloud depth information z is used. r To add color information to the binary image and address the instability of edge detection caused by the small number of laser point clouds, the data needs to be filtered. A preset filtering algorithm is used to set the points around the mapped pixels to the same color, and then an indirect method is used to extract the straight line segments of the laser point cloud.

[0090] Then, in the synchronous, same-scene laser point cloud and planar image, the edge data of the same reference object is determined, its offset is calculated, and based on the offsets of multiple sets of reference objects, the mapping equation between the radar coordinate system and the image coordinate system is corrected using the local offset.

[0091] Suppose the reference object has N laser points along its vertical edge. We can deduce that the pixel coordinates corresponding to each laser point cloud are (u1, v1), (u2, v2), ..., (u...). N ,v N The average horizontal coordinates of the pixels are:

[0092]

[0093] After determining the vertical edge of the reference object, the Hough algorithm is used to determine the longer edge, and M pixels (u) are selected. c_1 ,v c_1 ),(uc_2 ,v c_2 ),…,(u c_M ,v c_M The average horizontal coordinates of the pixels are:

[0094]

[0095] The corrected horizontal coordinate offset can then be obtained as:

[0096]

[0097] The corrected mapping equation is then obtained as follows:

[0098]

[0099] Example 3

[0100] Figure 3 This is a structural diagram of the data fusion device provided in Embodiment 3 of this application. This embodiment is applicable to the fusion of laser point clouds and planar images. The device can be implemented in software and / or hardware and can be configured in an electronic device. Figure 3 As shown, the data fusion device 300 may include: an original image acquisition module 310, a pixel coordinate transformation module 320, a transformation relationship determination module 330, and a data fusion module 340, wherein,

[0101] The original image acquisition module 310 is used to acquire the original image captured by the camera when the target object is illuminated by a laser point cloud; wherein the original image includes the target object;

[0102] The pixel coordinate determination module 320 is used to obtain the pixel coordinates based on the position of the laser point cloud in the original image;

[0103] The transformation relationship determination module 330 is used to obtain the transformation relationship between the radar coordinate system and the imaging coordinate system based on the pixel point cloud coordinates, the camera coordinate system, and the radar coordinate system.

[0104] The data fusion module 340 is used to fuse the laser point cloud and the original image according to the transformation relationship.

[0105] The technical solution of this application embodiment obtains the transformation relationship between the radar coordinate system and the imaging coordinate system by transforming the laser point cloud in the imaging coordinate system, camera coordinate system, and radar coordinate system. Based on this transformation relationship, the laser point cloud and the original image can be directly fused. The advantage of this approach is that it eliminates the need to convert the laser point cloud into a depth image before fusing it with the original image. Instead, it unifies the reference coordinate system through coordinate transformation, thereby reducing the difficulty of data fusion, reducing the time spent in the data fusion process, and improving the efficiency of fusing point cloud data and image data.

[0106] In one optional implementation, the conversion relationship determination module 330 may include:

[0107] The mapping relationship determination unit is used to determine the mapping relationship between the imaging coordinate system and the camera coordinate system based on the pixel point cloud coordinates and the camera coordinate system;

[0108] The rotation and translation relationship determination unit is used to determine the rotation and translation relationship between the camera coordinate system and the radar coordinate system based on the pose of the camera coordinate system and the pose of the radar coordinate system.

[0109] The transformation relationship determination unit is used to determine the transformation relationship between the radar coordinate system and the imaging coordinate system based on the mapping relationship and the rotation and translation relationship.

[0110] In one optional embodiment, the rotation-translation relationship determination unit may include:

[0111] The rotation and translation relationship determination sub-unit is used to determine the rotation and translation relationship between the camera coordinate system and the radar coordinate system based on the distance components and attitude vector between the two origins of the camera coordinate system and the radar coordinate system.

[0112] In one alternative embodiment, the pixel coordinate determination module 320 may include:

[0113] A perspective transformation unit is used to determine the mapping relationship between the imaging coordinate system and the camera coordinate system based on the perspective relationship between the pixel point cloud coordinates and the laser point cloud in the camera coordinate system. In an optional embodiment, the data fusion device 300 may further include:

[0114] The conversion relationship correction module is used to correct the conversion relationship based on the vertical reference line located on the target object.

[0115] In one optional implementation, the conversion relationship correction module may include:

[0116] The horizontal coordinate determination unit is used to determine the horizontal coordinates of each reference point cloud in the vertical reference line in the imaging coordinate system.

[0117] The reference line correction unit is used to correct the transformation relationship based on the average level coordinates, according to a preset straight line detection algorithm.

[0118] In one alternative embodiment, the data fusion device 300 may further include:

[0119] The binary image determination module is used to determine the binary image of the target object based on the transformation relationship;

[0120] The three-primary-color image acquisition module is used to obtain three-primary-color images based on point cloud depth information and binary images;

[0121] The reference line determination module is used to extract vertical reference lines from the three primary color images.

[0122] The data fusion apparatus provided in this application embodiment can execute the data fusion method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing each data fusion method.

[0123] Example 4

[0124] Figure 4 This is a structural diagram of an electronic device provided in Embodiment 4 of this application. Figure 4 A block diagram is shown of an exemplary electronic device 412 suitable for implementing embodiments of the present application. Figure 4 The electronic device 412 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0125] like Figure 4 As shown, electronic device 412 is represented in the form of a general-purpose computing device. The components of electronic device 412 may include, but are not limited to: one or more processors or processing units 416, system memory 428, and bus 418 connecting different system components (including system memory 428 and processing unit 416).

[0126] Bus 418 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0127] Electronic device 412 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 412, including volatile and non-volatile media, removable and non-removable media.

[0128] System memory 428 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 430 and / or cache memory 432. Electronic device 412 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 434 may be used to read and write non-removable, non-volatile magnetic media (… Figure 4 Not shown; commonly referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 418 via one or more data media interfaces. Memory 428 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0129] A program / utility 440 having a set (at least one) of program modules 442 may be stored, for example, in memory 428. Such program modules 442 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 442 typically perform the functions and / or methods described in the embodiments of this application.

[0130] Electronic device 412 can also communicate with one or more external devices 414 (e.g., keyboard, pointing device, display 424, etc.), and with one or more devices that enable a user to interact with electronic device 412, and / or with any device that enables electronic device 412 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 422. Furthermore, electronic device 412 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 420. As shown, network adapter 420 communicates with other modules of electronic device 412 via bus 418. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0131] The processing unit 416 executes various functional applications and data processing by running at least one of the multiple programs stored in the system memory 428, such as implementing the data fusion method provided in the embodiments of this application.

[0132] Example 5

[0133] Embodiment 5 of this application also provides a computer-readable storage medium storing a computer program (or computer-executable instructions) thereon. When executed by a processor, the program is used to perform the data fusion method provided in this application embodiment: acquiring an original image captured by a camera when a target object is illuminated by a laser point cloud; wherein the original image includes the target object; obtaining pixel point cloud coordinates based on the position of the laser point cloud in the original image; obtaining the transformation relationship between the radar coordinate system and the imaging coordinate system based on the pixel point cloud coordinates, the camera coordinate system, and the radar coordinate system; and fusing the laser point cloud and the original image according to the transformation relationship.

[0134] The computer storage medium in this application embodiment can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0135] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0136] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0137] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A data fusion method, characterized in that, include: Acquire the original image captured by the camera when the target object is illuminated by a laser point cloud; wherein the original image includes the target object and the laser point cloud projected onto the target object; The pixel point cloud coordinates are obtained based on the position coordinates of the laser point cloud in the original image; Based on the pixel point cloud coordinates, camera coordinate system, and radar coordinate system, the transformation relationship between the radar coordinate system and the imaging coordinate system is obtained; Based on the transformation relationship, the laser point cloud and the original image are fused; The step of obtaining the transformation relationship between the radar coordinate system and the imaging coordinate system based on the pixel point cloud coordinates, the camera coordinate system, and the radar coordinate system includes: Based on the pixel point cloud coordinates and the camera coordinate system, determine the mapping relationship between the imaging coordinate system and the camera coordinate system; Based on the pose of the camera coordinate system and the radar coordinate system, determine the rotation and translation relationship between the camera coordinate system and the radar coordinate system. Based on the mapping relationship and the rotation and translation relationship, determine the transformation relationship between the radar coordinate system and the imaging coordinate system; The step of determining the mapping relationship between the imaging coordinate system and the camera coordinate system based on the pixel point cloud coordinates and the camera coordinate system includes: Based on the pixel point cloud coordinates and the perspective relationship of the laser point cloud in the camera coordinate system, the mapping relationship between the imaging coordinate system and the camera coordinate system is determined.

2. The method according to claim 1, characterized in that, Determining the rotation and translation relationship between the camera coordinate system and the radar coordinate system based on the pose of the camera coordinate system and the radar coordinate system includes: Based on the distance components and attitude vectors between the two origins of the camera coordinate system and the radar coordinate system, the rotation and translation relationship between the camera coordinate system and the radar coordinate system is determined.

3. The method according to any one of claims 1-2, characterized in that, Before fusing the laser point cloud and the original image according to the transformation relationship, the method further includes: The transformation relationship is corrected based on the vertical reference line located on the target object.

4. The method according to claim 3, characterized in that, The step of correcting the transformation relationship based on a vertical reference line located on the target object includes: Determine the horizontal coordinates of each reference point cloud in the vertical reference line in the imaging coordinate system; Based on a preset straight line detection algorithm, the transformation relationship is corrected according to the average level coordinates.

5. The method according to claim 3, characterized in that, The method further includes: Based on the transformation relationship, the binary image of the target object is determined; Based on the point cloud depth information and the binary image, a three-color image is obtained; Extract the vertical reference line from the three primary color images.

6. A data fusion device, characterized in that, include: The original image acquisition module is used to acquire the original image captured by the camera when the target object is illuminated by a laser point cloud; wherein, the original image includes the target object and the laser point cloud projected on the target object; The pixel coordinate determination module is used to obtain the pixel point cloud coordinates based on the position coordinates of the laser point cloud in the original image; The transformation relationship determination module is used to obtain the transformation relationship between the radar coordinate system and the imaging coordinate system based on the pixel point cloud coordinates, the camera coordinate system, and the radar coordinate system. The data fusion module is used to fuse the laser point cloud and the original image according to the transformation relationship; The conversion relationship determination module includes: The mapping relationship determination unit is used to determine the mapping relationship between the imaging coordinate system and the camera coordinate system based on the pixel point cloud coordinates and the camera coordinate system; The rotation and translation relationship determination unit is used to determine the rotation and translation relationship between the camera coordinate system and the radar coordinate system based on the pose of the camera coordinate system and the pose of the radar coordinate system. The transformation relationship determination unit is used to determine the transformation relationship between the radar coordinate system and the imaging coordinate system based on the mapping relationship and the rotation and translation relationship; The mapping relationship determination unit includes: The perspective transformation unit is used to determine the mapping relationship between the imaging coordinate system and the camera coordinate system based on the perspective relationship between the pixel point cloud coordinates and the laser point cloud in the camera coordinate system.

7. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a data fusion method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a data fusion method as described in any one of claims 1-5.

Images