A fusion ranging method, apparatus, device and storage medium

Abstract

Embodiments of the present application disclose a kind of fusion ranging method, device, equipment and storage medium. Including: obtaining panoramic camera data and laser radar data;Panoramic camera data and laser radar data are aligned, and the target laser radar data corresponding to panoramic camera data is determined;According to the pre-configured camera calibration parameter set, restore panoramic camera data, determine the pixel point coordinate information corresponding to each pixel point in panoramic camera data;According to each pixel point coordinate information, camera calibration parameter set, pre-configured radar calibration parameter set and target laser radar data, generate fusion ranging result.The present application combines laser radar data and panoramic camera data, uses the way of first synchronous matching then restores split, without multiple calibration to vehicle-mounted camera, also without switching calibration parameter in the process of driving, reduce overall data operation amount, the ranging precision under non-planar scene determined by combining radar data and camera data is higher.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless measurement, in particular to a fusion ranging method, device, equipment and storage medium. BACKGROUND

[0002] With the development of the times, people's living quality is continuously improved, and cars gradually enter thousands of households and become one of the main means of transportation. While cars bring convenience to people's life and improve travel efficiency, the increasing number of vehicle ownership brings more parking demand, and the number of on-ground parking spaces has been difficult to meet the existing parking demand, and multi-layer underground parking lots have gradually become the mainstream means for cities to solve parking problems.

[0003] Vehicle assisted driving or automatic driving function needs to rely on the acquisition of surrounding environment information to construct a map, and then to plan a path according to the constructed map. However, when the existing vehicle-mounted camera performs inverse perspective mapping (IPM) calibration, it is assumed that the ground around the vehicle is a plane, which is not true on a spiral ramp, that is, the result of distance determination based on the parameters of plane calibration is inaccurate.

[0004] However, the multi-layer underground parking lot is realized by connecting the spiral channel or slope between the two layers when descending or ascending, and the existing vehicle often pre-calibrates the vehicle-mounted camera for different scenes when realizing wireless cross-layer ranging, and switches the calibration parameters according to the actual scene during driving to realize cross-layer ranging. However, the driving environment of the vehicle is different, and even if it is a spiral scene, the different inclination angles will also affect the measurement result, and the unified calibration parameters cannot perfectly adapt to the driving environment, resulting in a large error. And the switching of different types of calibration parameters is complex, and the data operation amount is high, which will greatly affect the driving experience of the user. SUMMARY

[0005] The present application provides a fusion ranging method, device, equipment and storage medium, which avoids multiple calibrations of the vehicle-mounted camera and switching of calibration parameters during driving, reduces the data operation amount, and improves the ranging accuracy in non-planar scenes.

[0006] In a first aspect, an embodiment of the present application provides a fusion ranging method, comprising:

[0007] obtaining panoramic camera data and laser radar data;

[0008] aligning the panoramic camera data and the laser radar data, and determining target laser radar data corresponding to the panoramic camera data;

[0009] restore the panoramic camera data according to the pre-configured camera calibration parameter set, and determine pixel point coordinate information corresponding to each pixel point in the panoramic camera data;

[0010] According to the pixel point coordinate information, the camera calibration parameter set, the pre-configured radar calibration parameter set, and the target laser radar data, depth information is supplemented for the panoramic camera data, and a fusion ranging result is generated.

[0011] In a second aspect, an embodiment of the present application further provides a fusion ranging device, comprising:

[0012] A data acquisition module is configured to acquire panoramic camera data and laser radar data.

[0013] A data alignment module is configured to perform data alignment on the panoramic camera data and the laser radar data, and determine target laser radar data corresponding to the panoramic camera data.

[0014] A coordinate information determination module is configured to restore the panoramic camera data according to the pre-configured camera calibration parameter set, and determine pixel point coordinate information corresponding to each pixel point in the panoramic camera data.

[0015] A ranging result determination module is configured to supplement depth information for the panoramic camera data according to the pixel point coordinate information, the camera calibration parameter set, the pre-configured radar calibration parameter set, and the target laser radar data, and generate a fusion ranging result.

[0016] In a third aspect, an embodiment of the present application further provides a fusion ranging device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the fusion ranging method of any embodiment of the present application.

[0020] In a fourth aspect, an embodiment of the present application further provides a storage medium containing computer executable instructions, which, when executed by a computer processor, enable the computer processor to execute the fusion ranging method of any embodiment of the present application.

[0021] The embodiment of the present application provides a kind of fusion ranging method, device and equipment and storage medium, by obtaining panoramic camera data and laser radar data;Panoramic camera data and laser radar data are aligned, and the target laser radar data corresponding to panoramic camera data is determined;According to the pre-configured camera calibration parameter set, restore panoramic camera data, determine the pixel point coordinate information corresponding to each pixel point in panoramic camera data;According to each pixel point coordinate information, camera calibration parameter set, pre-configured radar calibration parameter set and target laser radar data, supplement depth information for panoramic camera data, generate fusion ranging result.The above technical scheme is used, laser radar data and panoramic camera data are combined, and the way of first synchronous matching and then restoring and splitting is used, so that panoramic camera data and laser radar data only need to be matched once, that is, the depth information of camera data can be added, and the matching accuracy of radar data and camera data is higher.Because the depth information is provided by laser radar data, it is not necessary to calibrate vehicle-mounted camera for different use scenarios multiple times, and it is not necessary to switch calibration parameters during driving, so as to reduce the overall data operation amount.And because the specific calibration parameters required in different non-planar scenes are different, if the vehicle-mounted camera is calibrated in advance, the calibration parameters suitable for all scenes cannot be obtained, and the depth information collected by laser radar data is real-time depth information, so that the ranging accuracy in non-planar scene determined by combining radar data and camera data is higher.

[0022] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1 The flow chart of a fusion ranging method provided for the first embodiment of the present application is shown in the figure.

[0025] Figure 2 The flow chart of another fusion ranging method provided for the second embodiment of the present application is shown in the figure.

[0026] Figure 3A flowchart of a fusion ranging method provided by the embodiment one of the present application, the embodiment can be applicable to the case of measuring the distance between the vehicle and the external obstacles in the driving environment during driving, the method can be executed by a fusion ranging device, the fusion ranging device can be realized in the form of hardware and / or software, the fusion ranging device can be configured in a fusion ranging equipment, the fusion ranging equipment can be a notebook, a desktop computer, a smart tablet or a vehicle, etc., and the embodiment of the present application does not limit this.

[0027] Figure 4 A structural schematic diagram of a fusion ranging device provided by the embodiment three of the present application.

[0028] Figure 5 A structural schematic diagram of a fusion ranging device provided by the embodiment four of the present application. DETAILED DESCRIPTION

[0029] In order to enable the person skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment one

[0032] Figure 1 A flowchart of a fusion ranging method provided by the embodiment one of the present application, the embodiment can be applicable to the case of measuring the distance between the vehicle and the external obstacles in the driving environment during driving, the method can be executed by a fusion ranging device, the fusion ranging device can be realized in the form of hardware and / or software, the fusion ranging device can be configured in a fusion ranging equipment, the fusion ranging equipment can be a notebook, a desktop computer, a smart tablet or a vehicle, etc., and the embodiment of the present application does not limit this.

[0033] As Figure 1As shown, the fusion ranging method provided by the embodiment of the application specifically comprises the following steps:

[0034] S101, acquire panoramic camera data and laser radar data.

[0035] In this embodiment, the panoramic camera data can be specifically understood as the overhead view of the 360° range around the vehicle collected during driving, wherein the information contained in each pixel point is two-dimensional information. The laser radar data can be specifically understood as the information of the space points around the vehicle collected during driving, wherein the information contained in each pixel point is three-dimensional information.

[0036] Specifically, during driving of the vehicle, the vehicle-mounted camera and the laser radar arranged at the fixed position of the vehicle are used to collect the environment information around the vehicle, and the image data and the radar data in the collection range of the vehicle-mounted camera and the laser radar are obtained. The data of each vehicle-mounted camera and laser radar is spliced according to the arranged position, and the panoramic camera data and the laser radar data in the 360° range around the vehicle are obtained.

[0037] S102, align the panoramic camera data and the laser radar data, and determine the target laser radar data corresponding to the panoramic camera data.

[0038] In this embodiment, the target laser radar data can be specifically understood as the radar data that is most matched with the currently collected panoramic camera data, or the radar data whose collected information is most similar to the information contained in the panoramic camera data.

[0039] Specifically, since depth information needs to be added to the acquired camera data when ranging, so as to convert the two-dimensional data into three-dimensional data, the laser radar data overlapping with the collection range of the panoramic camera data needs to be acquired. Since the laser radar data is collected in real time, the camera data and the radar data collected at the same time should be aimed at the same environment around the vehicle, so the panoramic camera data and the laser radar data can be aligned according to the collection time, and the data points most matched with the panoramic camera data are screened from the laser radar data. The set of the screened data points is determined as the target laser radar data.

[0040] In the embodiment of the application, since the collection range of the camera for collecting the panoramic camera data and the laser radar for collecting the laser radar data may not be completely consistent, the alignment of the data in one camera may need to input multiple radar data intersecting with the camera range, which brings a large amount of calculation. In the embodiment of the application, the alignment of the camera data and the radar data in the vehicle surround view range is solved at one time by aligning the panoramic camera data and the laser radar data, which reduces the calculation amount, improves the calculation speed, and further improves the subsequent ranging accuracy.

[0041] S103, restore the panoramic camera data according to the pre-configured camera calibration parameter set, and determine the pixel point coordinate information corresponding to each pixel point in the panoramic camera data.

[0042] In the embodiment, the camera calibration parameter set can be understood as a set of camera calibration parameters obtained by calibrating each vehicle-mounted camera in a planar scene and arranged on the vehicle. It can be understood that the camera calibration parameters of different vehicle-mounted cameras can be represented in a group form, and can be marked with the number of the corresponding vehicle-mounted camera or other information representing the identity of the camera. The pixel point coordinate information can be understood as the position information of each pixel point in the corresponding camera in the panoramic camera data.

[0043] Specifically, since the camera calibration parameters of the vehicle-mounted camera have been pre-configured as the camera calibration parameter set, after obtaining the panoramic camera data, the camera data collected by each vehicle-mounted camera in the panoramic camera data can be determined according to the setting position of the vehicle-mounted camera on the vehicle, and then the camera data is restored by the camera calibration parameters corresponding to the vehicle-mounted camera in the camera calibration parameter set, to obtain the position information of each pixel point in the camera data relative to the vehicle-mounted camera, and then the pixel point coordinate information of all pixel points in the panoramic camera data is determined.

[0044] S104, according to the pixel point coordinate information, the camera calibration parameter set, the pre-configured radar calibration parameter set and the target laser radar data, supplement the depth information for the panoramic camera data, and generate the fusion ranging result.

[0045] In the embodiment, the radar calibration parameter set can be understood as a set of radar calibration parameters obtained by calibrating each laser radar based on the relative position of the vehicle and arranged on the vehicle.

[0046] Specifically, according to the pixel point coordinate information, the camera calibration parameter set, the radar calibration parameter set and the target laser radar data, the corresponding relationship between each pixel point in the target laser radar data and the pixel point coordinate information is determined. Since each pixel point in the target laser radar data has depth information, after the corresponding relationship is determined, the depth information of each pixel point in the target laser radar data can be supplemented to the corresponding pixel point in the panoramic camera data, to obtain the fusion ranging result according to the distance information of the three-dimensional pixel point relative to the vehicle after supplementing the depth.

[0047] The technical scheme of the embodiment is as follows: panoramic camera data and laser radar data are acquired; the panoramic camera data and the laser radar data are aligned, target laser radar data corresponding to the panoramic camera data is determined; the panoramic camera data is restored according to a preconfigured camera calibration parameter set, and pixel point coordinate information corresponding to each pixel point in the panoramic camera data is determined; depth information is supplemented for the panoramic camera data according to the pixel point coordinate information, the camera calibration parameter set, a preconfigured radar calibration parameter set and the target laser radar data, and a fusion ranging result is generated. By using the above technical scheme, the laser radar data and the panoramic camera data are combined, a synchronous matching and then a restoration and splitting mode is used, so that the panoramic camera data and the laser radar data only need to be matched once, the depth information of the camera data can be added, and the matching accuracy of the radar data and the camera data is higher. Since the depth information is provided by the laser radar data, the vehicle-mounted camera does not need to be calibrated for different use scenarios multiple times, and the calibration parameters do not need to be switched during driving, so that the overall data operation amount is reduced. Since the specific calibration parameters required in different non-planar scenes are different, the calibration of the vehicle-mounted camera in advance cannot obtain calibration parameters suitable for all scenes, and the depth information collected by the laser radar data is real-time depth information, so that the ranging accuracy in the non-planar scene determined by combining the radar data and the camera data is higher.

[0048] Embodiment Two

[0049] Figure 2 The flowchart of another fusion ranging method provided by Embodiment Two of the application is based on the optimization of the above-mentioned optional technical schemes. The correspondence between the panoramic camera data and the laser radar data is completed by using the time stamps in the panoramic camera data and the laser radar data, the target laser radar data that is accurately matched is determined, the method of restoring the panoramic camera data is determined, the mapping between the radar pixel points and the camera pixel points is realized according to the coordinate transformation matrix between each camera coordinate system and each radar coordinate system after the panoramic camera data is restored, the depth information addition for the camera pixel points is realized, the corresponding fusion ranging result is obtained, the overall operation amount is reduced, the matching accuracy is improved, and the ranging accuracy in the non-planar scene determined by combining the radar data and the camera data is higher.

[0050] As shown in Figure 2 , the fusion ranging method provided by Embodiment Two of the application specifically includes the following steps:

[0051] S201, acquiring panoramic camera data and laser radar data.

[0052] Further, to ensure the correct implementation of the fusion ranging method, before acquiring the panoramic camera data and the laser radar data, the method further comprises: determining camera calibration parameters of each surround-view camera and radar calibration parameters of each laser radar according to the configuration positions of the at least two surround-view cameras and the at least two laser radars; calibrating each surround-view camera according to the camera calibration parameters to generate and store a camera calibration parameter set; and calibrating each laser radar according to the radar calibration parameters to generate and store a radar calibration parameter set.

[0053] Specifically, at least two surround-view cameras and at least two laser radars can be respectively arranged at different positions of the vehicle to realize the collection of image data and spatial information within a 360° range around the vehicle. Since the arrangement positions of the different surround-view cameras and laser radars are different, the above devices need to be calibrated first to facilitate the subsequent processing of the data collected by each surround-view camera and each laser radar. Optionally, the vehicle can be driven onto a large plane checkerboard, and the calibration of each surround-view camera and each laser radar can be completed based on the configuration positions of the surround-view cameras and the laser radars relative to the vehicle. Then, the camera calibration parameters of each surround-view camera and the radar calibration parameters of each laser radar can be obtained, and a camera calibration parameter set composed of the camera calibration parameters and a radar calibration parameter set composed of the radar calibration parameters can be stored for subsequent data processing.

[0054] Further, before acquiring the panoramic camera data and the laser radar data, the method further comprises: acquiring single-side camera data collected by the at least two surround-view cameras; and splicing the single-side camera data according to the camera calibration parameters of the surround-view cameras to determine the panoramic camera data.

[0055] In this embodiment, the single-side camera data can be understood as data collected by one surround-view camera and facing in one direction.

[0056] Specifically, since the camera calibration parameters of each surround-view camera contain the position information of the surround-view camera in the vehicle, the adjacent relationship between the surround-view cameras can be determined, the single-side camera data collected by each adjacent surround-view camera can be spliced according to the relative position relationship between the camera and the vehicle, and the spliced camera data can be determined as the panoramic camera data.

[0057] S202, determine a camera timestamp corresponding to the panoramic camera data.

[0058] In this embodiment, the camera timestamp can be understood as a digital signature tag carried in the panoramic camera data and used to indicate the collection time of the panoramic camera data.

[0059] Specifically, when each single-side camera data constituting the panoramic camera data is collected, a timestamp corresponding to the data collection time is stored in the camera data based on the digital signature technology, and each single-side camera data constituting the panoramic camera data is collected at the same time, that is, the data corresponding to each pixel point in the panoramic camera data has the same camera timestamp.

[0060] S203, the radar point cloud data with the closest timestamp to the camera timestamp in the laser radar data is determined as the target laser radar data corresponding to the panoramic camera data.

[0061] Specifically, like the process of collecting single-side camera data, when the laser radar data is collected by different laser radars, each laser radar data is also time-stamped according to the collection time, and since the laser radar data is collected in real time, that is, continuous time data is collected, it can be considered that the laser radar data in a small period of time closest to the current time is saved in a queue. Assuming that the camera timestamp is t1, the laser radar data closest to the time t1 can be found in the laser radar data queue using t1, and the found laser radar data is determined as the target laser radar data corresponding to the panoramic camera data.

[0062] S204, the panoramic camera data is divided according to the pre-configured camera calibration parameter set, and the panoramic camera data corresponding to each camera calibration parameter is determined as single-side camera data.

[0063] Specifically, according to the setting direction of each camera calibration parameter in the pre-configured camera calibration parameter set on the vehicle, the camera calibration parameter corresponding to each part of the camera data in the panoramic camera data is determined, and the single-side camera data corresponding to each surround-view camera is obtained by coordinate restoration of the panoramic camera data in the corresponding setting direction of each camera calibration parameter.

[0064] S205, the camera number corresponding to each pixel point in the panoramic camera data is determined according to each camera calibration parameter.

[0065] Among them, each pixel point corresponding to the single-side camera data has the same camera number.

[0066] Optionally, when calibrating each surround-view camera in the vehicle, different camera numbers can be configured for different surround-view cameras, and the camera number is stored in association with the camera calibration parameter in the camera calibration parameter set. Since the panoramic camera data can be divided into multiple groups of single-side camera data corresponding to each surround-view camera according to each camera calibration parameter, each group of single-side camera data has multiple pixel points, and the camera numbers corresponding to each pixel point in the same group of single-side camera data are consistent, so the camera number of the surround-view camera corresponding to each pixel in the panoramic camera data can be determined.

[0067] For example, assuming that the panoramic camera data corresponds to four regions, i.e., front, rear, left and right, which correspond to four surround view cameras respectively, the camera number of each pixel point in the front region of the panoramic camera data can be set as the camera number corresponding to the front surround view camera.

[0068] In S206, the single-side camera coordinate information of each pixel point corresponding to each single-side camera data is determined by respectively performing coordinate restoration on each pixel point corresponding to each single-side camera data according to the camera calibration parameters.

[0069] Specifically, for each set of single-side camera data, since it is part of the panoramic camera data, i.e., the top view data obtained after inverse perspective transformation, and since the setting positions of the surround view cameras in the vehicle are different, the camera calibration parameters required for converting the original single-side camera data collected by the surround view cameras into top view data are different, therefore, the single-side camera data converted into top view data can also be inversely restored according to the camera calibration parameters to obtain the coordinate information of each pixel point in the original image collected by the surround view cameras, which is determined as the single-side camera coordinate information of the pixel point.

[0070] In S207, the camera number and the single-side camera coordinate information are combined to determine the pixel point coordinate information of each pixel point corresponding to the single-side camera data.

[0071] Specifically, for each pixel point in the single-side camera data corresponding to each surround view camera, the camera number of the surround view camera and the single-side camera coordinate information corresponding to the pixel point are combined to determine the pixel point coordinate information corresponding to the pixel point.

[0072] In S208, the camera coordinate system set is determined according to the camera calibration parameter set, and the radar coordinate system set is determined according to the pre-configured radar calibration parameter set.

[0073] In this embodiment, the camera coordinate system can be specifically understood as a three-dimensional rectangular coordinate system with the focus center of the surround view camera as the origin and the optical axis as the Z axis. It can be understood that the determination of the camera coordinate system is related to the position of the surround view camera relative to the vehicle, i.e., it can be determined according to the camera calibration parameters. The radar coordinate system can be specifically understood as a right-handed screw coordinate system with the earth center as the coordinate origin, the north pole as the Z axis, and the meridian plane with longitude 0 as the XOZ plane, i.e., it can be understood as a polar coordinate system used to indicate the spatial position relationship of the radar collected data.

[0074] Specifically, a camera coordinate system corresponding to each surround-view camera is determined according to camera calibration parameters of the surround-view cameras relative to the vehicle, and a camera coordinate system set composed of the camera coordinate systems is obtained. A radar coordinate system corresponding to each lidar is determined according to radar calibration parameters of the lidars relative to the vehicle, and a radar coordinate system set composed of the radar coordinate systems is obtained.

[0075] S209, each pixel point coordinate information located in the range of the target lidar data is determined as internal pixel point coordinate information, and other pixel point coordinate information is determined as external pixel point coordinate information.

[0076] Specifically, since the range of data collected by the lidar in space is different from the range of the surround-view camera, that is, the target lidar data cannot be one-to-one corresponding to the surround-view camera data, at this time, the pixel points located in the range of each lidar can be determined as internal pixel points, that is, each pixel point coordinate information located in the range of each lidar in the panoramic camera data is determined as internal pixel point information, and other pixel point coordinate information beyond the range of each lidar is determined as external pixel point coordinate information.

[0077] S210, according to the coordinate transformation relationship between each camera coordinate system and each radar coordinate system, a target lidar point corresponding to each internal pixel point coordinate information is determined from the target lidar data, and the depth information in each target lidar point is added to the corresponding internal pixel point coordinate information.

[0078] In this embodiment, the target lidar point can be understood as a pixel point in the target lidar data closest to the pixel point position corresponding to the internal pixel point coordinate information.

[0079] Specifically, since the representation of the pixel point position in each camera coordinate system and each radar coordinate system is not unified, before the depth information mapping and adding, the coordinate conversion relationship between each camera coordinate system and each radar coordinate system needs to be determined to realize the mapping of the pixel point in the lidar data to the pixel point in the camera coordinate system of different surround-view cameras. Since the target lidar data is the lidar data that has completed alignment with the panoramic camera data, and the monocular camera data in the range corresponding to each surround-view camera can be divided according to the camera calibration parameters, and then the target lidar data corresponding to the position of the monocular camera data can be screened from the target lidar data, based on the coordinate transformation relationship corresponding to the surround-view camera, the coordinate information of the target lidar data in the camera coordinate system can be obtained, and then the radar point closest to each internal pixel point coordinate information can be determined as the target lidar point. The similar radar point and the camera pixel point can be considered as the pixel points with the same position in space, and accordingly the depth information in the target lidar point can be added to the corresponding internal pixel point coordinate information.

[0080] Further, Figure 3 A flowchart example of adding depth information of each target laser radar point to corresponding internal pixel point coordinate information according to the coordinate transformation relationship between each camera coordinate system and each radar coordinate system is provided for the second embodiment of the present application, as shown in Figure 3 The specific steps include the following steps:

[0081] S2101, according to the coordinate transformation relationship between each camera coordinate system and each radar coordinate system, a transformation matrix set is constructed.

[0082] Optionally, the transformation matrix is constructed as follows:

[0083]

[0084] wherein, is the representation of the laser radar in the vehicle body coordinate system, is the representation of the surround-view camera in the vehicle body coordinate system, is the transformation matrix of the laser radar to the surround-view camera.

[0085] It can be understood that there are different transformation matrices between the coordinate systems corresponding to different surround-view cameras and different laser radars. The above formula is only an example of the transformation matrix determination method of the camera coordinate system of one surround-view camera and the radar coordinate system of one laser radar.

[0086] S2102, according to each transformation matrix, the target laser radar data is mapped to determine the one-sided radar data corresponding to each camera coordinate system.

[0087] In this embodiment, the one-sided radar data can be understood as the radar data corresponding to the data collection range of each surround-view camera in the target laser radar data, which is obtained after the transformation matrix mapping in the camera coordinate system. The mapping method is as follows:

[0088]

[0089] wherein, represents the laser radar data in the radar coordinate system, represents the laser radar data in the camera coordinate system after transformation by the transformation matrix.

[0090] S2103, for each internal pixel point coordinate information, according to the camera number in the internal pixel point coordinate information, the target camera coordinate system is determined, and the one-sided radar data corresponding to the target camera coordinate system is determined as the target one-sided radar data.

[0091] Specifically, for each internal pixel point coordinate information, since the camera number information is contained therein, the surround view camera corresponding to the internal pixel point coordinate information can be determined, that is, the camera coordinate system corresponding to the surround view camera can be determined as the target camera coordinate system corresponding to the pixel point, and then the one-sided radar data corresponding to the target camera coordinate system can be determined as the target one-sided radar data used for matching the corresponding camera data in the target camera coordinate system.

[0092] S2104, interpolate the target one-sided radar data to determine the target laser radar point corresponding to the internal pixel point coordinate information.

[0093] Specifically, since the target one-sided radar data corresponds to the pixel point in the target camera coordinate system, the pixel points corresponding to each internal pixel point coordinate information in the target camera coordinate system do not completely correspond, and the target one-sided radar data can be more sparse, at this time, the target one-sided radar data can be interpolated to obtain the target laser radar point closest to the pixel point corresponding to the internal pixel point coordinate information.

[0094] S211, according to the relative position relationship between each external pixel point coordinate information and internal pixel point coordinate information, add the relative depth information to the corresponding external pixel point coordinate information, and determine the panoramic camera data after the depth information is supplemented as the fusion ranging result.

[0095] Specifically, for each external pixel point coordinate information corresponding pixel point not located in the range of the target laser radar data, the depth information contained in the target laser radar data cannot be directly added to each external pixel point coordinate information, and if the depth information is added to each external pixel point coordinate information according to interpolation and forward modeling, the accuracy of the determined depth information will be low, which will affect the determination of the subsequent fusion ranging result. At this time, according to the relative position relationship between the pixel point corresponding to each external pixel point coordinate information and the internal pixel point coordinate information, the relative depth information between the external pixel point and the internal pixel point is determined, the relative depth information is added to the corresponding external pixel point coordinate information, so that the depth information addition in the internal pixel point coordinate information and the external pixel point coordinate information is completed, that is, the panoramic camera data after the depth information is supplemented is obtained, and then the distance information between two pixel points in the panoramic camera data containing three-dimensional information can be determined, and the distance information of each pixel point relative to the vehicle can also be determined, to generate the corresponding fusion ranging result.

[0096] The technical scheme of the embodiment is realized by matching the panoramic camera data and the laser radar data through the time stamps, determining the target laser radar data that is accurately matched, and based on the matched target laser radar data, the restored panoramic camera data, the transformation matrix between the camera coordinate system corresponding to the surround view camera included in the vehicle and the radar coordinate system corresponding to the laser radar, realizing the mapping between the radar pixel points and the camera pixel points, and adding the depth information for the camera pixel points, the added depth information is divided into the mapping addition of the direct depth information and the addition of the relative depth information, the accuracy of the three-dimensional coordinates in the panoramic camera data after the addition of the depth information is improved, the overall operation amount is reduced, the matching accuracy is improved, and the ranging accuracy in the non-planar scene determined by combining the radar data and the camera data is higher.

[0097] Embodiment three

[0098] Figure 4 A structure diagram of a fusion ranging device is provided for the third embodiment of the application, and the fusion ranging device comprises a data acquisition module 31, a data alignment module 32, a coordinate information determination module 33 and a ranging result determination module 34.

[0099] The data acquisition module 31 is configured to acquire panoramic camera data and laser radar data; the data alignment module 32 is configured to align the panoramic camera data and the laser radar data, and determine target laser radar data corresponding to the panoramic camera data; the coordinate information determination module 33 is configured to restore the panoramic camera data according to a preconfigured camera calibration parameter set, and determine pixel point coordinate information corresponding to each pixel point in the panoramic camera data; and the ranging result determination module 34 is configured to supplement depth information for the panoramic camera data according to the pixel point coordinate information, the camera calibration parameter set, a preconfigured radar calibration parameter set and the target laser radar data, and generate a fusion ranging result.

[0100] The technical scheme of the embodiment combines the laser radar data and the panoramic camera data, adopts the mode of synchronous matching and then restoration and splitting, so that the panoramic camera data and the laser radar data only need to be matched once, the depth information of the camera data can be added, and the matching accuracy of the radar data and the camera data is higher. Since the depth information is provided by the laser radar data, the vehicle-mounted camera does not need to be calibrated for different use scenarios multiple times, and the calibration parameters do not need to be switched during driving, so that the overall data operation amount is reduced. Since the specific calibration parameters required in different non-planar scenes are different, if the vehicle-mounted camera is calibrated in advance, the calibration parameters suitable for all scenes cannot be obtained, the depth information collected by the laser radar data is real-time depth information, so that the ranging accuracy in the non-planar scene determined by combining the radar data and the camera data is higher.

[0101] Optionally, the data alignment module 32 comprises:

[0102] a timestamp determination unit configured to determine a camera timestamp corresponding to the panoramic camera data.

[0103] a target radar data determination unit configured to determine, as target lidar data corresponding to the panoramic camera data, radar point cloud data in the lidar data having a timestamp closest to the camera timestamp.

[0104] Optionally, the coordinate information determination module 33 comprises:

[0105] a camera data division unit configured to divide the panoramic camera data according to a preconfigured set of camera calibration parameters, and determine, as single-sided camera data, panoramic camera data corresponding to each camera calibration parameter.

[0106] a camera number determination unit configured to determine, according to each camera calibration parameter, a camera number corresponding to each pixel point in the panoramic camera data; wherein each pixel point corresponding to the single-sided camera data has the same camera number.

[0107] a single-sided coordinate determination unit configured to restore, through each camera calibration parameter, the coordinates of each pixel point corresponding to each single-sided camera data, to determine single-sided camera coordinate information of each pixel point corresponding to each single-sided camera data.

[0108] a coordinate information determination unit configured to combine the camera number and the single-sided camera coordinate information to determine pixel point coordinate information of each pixel point corresponding to the single-sided camera data.

[0109] Optionally, the ranging result determination module 34 comprises:

[0110] a coordinate system set determination unit configured to determine, according to the set of camera calibration parameters, a set of camera coordinate systems, and determine, according to a preconfigured set of radar calibration parameters, a set of radar coordinate systems.

[0111] a pixel point range division unit configured to determine, as internal pixel point coordinate information, each pixel point coordinate information located in a range of the target lidar data, and determine, as external pixel point coordinate information, other pixel point coordinate information.

[0112] a depth information adding unit configured to determine, according to a coordinate transformation relationship between each camera coordinate system and each radar coordinate system, a target lidar point corresponding to each internal pixel point coordinate information from the target lidar data, and add depth information in each target lidar point to the corresponding internal pixel point coordinate information.

[0113] The ranging result determination unit is configured to add the relative depth information to the corresponding external pixel point coordinate information according to the relative position relationship between the external pixel point coordinate information and the internal pixel point coordinate information, and determine the panoramic camera data after the depth information is supplemented as the fusion ranging result.

[0114] Optionally, the depth information adding unit is specifically configured to:

[0115] According to the coordinate transformation relationship between the camera coordinate systems and the radar coordinate systems, a transformation matrix set is constructed.

[0116] According to the target laser radar data, the one-side radar data corresponding to the camera coordinate systems is determined.

[0117] For each internal pixel point coordinate information, the target camera coordinate system is determined according to the camera number in the internal pixel point coordinate information, and the one-side radar data corresponding to the target camera coordinate system is determined as the target one-side radar data.

[0118] The target one-side radar data is interpolated to determine the target laser radar point corresponding to the internal pixel point coordinate information.

[0119] Optionally, the fusion ranging device further comprises a calibration parameter determination module configured to determine camera calibration parameters of each ring-view camera and radar calibration parameters of each laser radar according to the configuration positions of the at least two ring-view cameras and the at least two laser radars before the panoramic camera data and the laser radar data are acquired, calibrate each ring-view camera by the camera calibration parameters to generate and store a camera calibration parameter set, and calibrate each laser radar by the radar calibration parameters to generate and store a radar calibration parameter set.

[0120] Optionally, the fusion ranging device further comprises a camera data splicing module configured to acquire one-side camera data collected by the at least two ring-view cameras before the panoramic camera data and the laser radar data are acquired, splice the one-side camera data according to the camera calibration parameters of each ring-view camera to determine the panoramic camera data.

[0121] The fusion ranging device provided by the embodiments of the present application can execute the fusion ranging method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0122] Embodiment Four

[0123] Figure 5A structure diagram of a fusion ranging device is provided for Embodiment Four of the present application. The fusion ranging device 40 can be an electronic device, intended to represent a wide variety of form factors of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent a wide variety of form factors of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the application as described and / or claimed herein.

[0124] As shown in Figure 5 The fusion ranging device 40 includes at least one processor 41, and a memory, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., connected with the at least one processor 41 in communication, where the memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 42 or loaded from the storage unit 48 into the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the fusion ranging device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected with each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0125] A plurality of components in the fusion ranging device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the fusion ranging device 40 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0126] The processor 41 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs various methods and processes described above, such as the fusion ranging method.

[0127] In some embodiments, the fusion ranging method can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 48. In some embodiments, parts or all of the computer program can be loaded and / or installed onto fusion ranging device 40 via, e.g., ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the fusion ranging method described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the fusion ranging method by other means, e.g., with the aid of firmware.

[0128] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0129] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0130] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0132] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0133] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0134] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0135] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A fusion ranging method, characterized in that, include: Acquire single-side camera data from at least two surround-view cameras; Based on the camera calibration parameters of each of the panoramic cameras, the data from each of the single-sided cameras are stitched together to determine the panoramic camera data; Acquire panoramic camera data and LiDAR data; Determine the camera timestamp corresponding to the panoramic camera data; The radar point cloud data whose timestamp is closest to the camera timestamp in the lidar data is identified as the target lidar data corresponding to the panoramic camera data; The panoramic camera data is reconstructed based on a pre-configured set of camera calibration parameters, and the pixel coordinate information corresponding to each pixel in the panoramic camera data is determined. The camera coordinate system set is determined based on the camera calibration parameter set, and the radar coordinate system set is determined based on the pre-configured radar calibration parameter set. The coordinate information of each pixel point located within the range of the target lidar data is determined as the internal pixel point coordinate information, and the coordinate information of other pixel points is determined as the external pixel point coordinate information. Based on the coordinate transformation relationship between each camera coordinate system and each radar coordinate system, the target lidar point corresponding to the coordinate information of each internal pixel point is determined from the target lidar data, and the depth information in each target lidar point is added to the corresponding internal pixel point coordinate information. Based on the relative positional relationship between the coordinates of each external pixel and the coordinates of the internal pixel, the relative depth information is added to the corresponding external pixel coordinate information, and the panoramic camera data after the depth information is supplemented is determined as the fusion ranging result.

2. The method according to claim 1, characterized in that, The step of reconstructing the panoramic camera data based on a pre-configured set of camera calibration parameters and determining the pixel coordinate information corresponding to each pixel in the panoramic camera data includes: The panoramic camera data is divided according to the pre-configured set of camera calibration parameters, and the panoramic camera data corresponding to each camera calibration parameter is determined as single-side camera data. The camera number corresponding to each pixel in the panoramic camera data is determined according to the calibration parameters of each camera; wherein, each pixel corresponding to the single-sided camera data has the same camera number; By using the camera calibration parameters of each camera, the coordinates of each pixel corresponding to each single-side camera data are restored, and the single-side camera coordinate information corresponding to each pixel of each single-side camera data is determined. The camera number is combined with the coordinate information of the single-sided camera to determine the pixel coordinate information of each pixel corresponding to the single-sided camera data.

3. The method according to claim 1, characterized in that, The step of determining the target lidar point corresponding to the coordinate information of each internal pixel point from the target lidar data based on the coordinate transformation relationship between each camera coordinate system and each lidar coordinate system includes: Based on the coordinate transformation relationship between each camera coordinate system and each radar coordinate system, construct a set of transformation matrices; The target lidar data is mapped according to each transformation matrix to determine the single-sided lidar data corresponding to each camera coordinate system. For each internal pixel coordinate information, the target camera coordinate system is determined according to the camera number in the internal pixel coordinate information, and the single-sided radar data corresponding to the target camera coordinate system is determined as the target single-sided radar data. Interpolate the target's single-sided radar data to determine the target lidar point corresponding to the internal pixel coordinate information.

4. The method according to any one of claims 1-3, characterized in that, Before acquiring panoramic camera data and LiDAR data, the following steps are also included: Based on the configuration positions of at least two surround-view cameras and at least two lidars, determine the camera calibration parameters of each surround-view camera and the lidar calibration parameters of each lidar. Each of the aforementioned camera calibration parameters is used to calibrate the surrounding camera, and a set of camera calibration parameters is generated and stored. Each lidar is calibrated using the corresponding radar calibration parameters, and a set of radar calibration parameters is generated and stored.

5. A fusion ranging device, characterized in that, include: The data acquisition module is used to acquire panoramic camera data and LiDAR data; A data alignment module is used to align the panoramic camera data and the lidar data to determine the target lidar data corresponding to the panoramic camera data. The coordinate information determination module is used to reconstruct the panoramic camera data based on a pre-configured set of camera calibration parameters and determine the pixel coordinate information corresponding to each pixel in the panoramic camera data. The ranging result determination module is used to supplement the panoramic camera data with depth information based on the coordinate information of each pixel point, the camera calibration parameter set, the pre-configured radar calibration parameter set, and the target lidar data, and generate a fused ranging result. The fusion ranging device includes: The camera data stitching module is used to acquire single-side camera data collected by at least two surround-view cameras before acquiring panoramic camera data and LiDAR data; and to stitch together the single-side camera data according to the camera calibration parameters of each surround-view camera to determine the panoramic camera data. The data alignment module includes: The timestamp determination unit is used to determine the camera timestamp corresponding to the panoramic camera data; The target radar data determination unit is used to determine the radar point cloud data in the lidar data whose timestamp is closest to the camera timestamp as the target lidar data corresponding to the panoramic camera data; The ranging result determination module includes: The coordinate system set determination unit is used to determine the camera coordinate system set according to the camera calibration parameter set and to determine the radar coordinate system set according to the pre-configured radar calibration parameter set. A pixel range division unit is used to determine the coordinate information of each pixel located within the range of the target lidar data as internal pixel coordinate information, and to determine the coordinate information of other pixels as external pixel coordinate information. The depth information addition unit is used to determine the target lidar point corresponding to the coordinate information of each internal pixel point from the target lidar data according to the coordinate transformation relationship between each camera coordinate system and each radar coordinate system, and to add the depth information of each target lidar point to the corresponding internal pixel point coordinate information. The ranging result determination unit is used to add relative depth information to the corresponding external pixel coordinate information according to the relative positional relationship between the coordinate information of each external pixel and the coordinate information of the internal pixel, and determine the panoramic camera data after the depth information is supplemented as the fused ranging result.

6. A fusion ranging device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the fusion ranging method according to any one of claims 1-4.

7. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the fusion ranging method as described in any one of claims 1-4.

Images