Cabin three-dimensional modeling method, device and equipment, storage medium and program product

Abstract

The invention provides a cabin three-dimensional modeling method, device and equipment, a storage medium and a program product, and relates to the technical field of three-dimensional modeling. The method comprises the following steps: determining a plurality of coordinates under a vehicle coordinate system based on a plurality of distance data measured by a distance measuring device; determining a rotation matrix of the image acquisition device relative to the vehicle coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system and a conversion relation between the vehicle coordinate system and the calibration device coordinate system; based on the translation vector of the image acquisition device relative to the vehicle coordinate system and the rotation matrix of the image acquisition device relative to the vehicle coordinate system, converting the plurality of coordinates under the vehicle coordinate system into a plurality of coordinates under the coordinate system of the image acquisition device; and constructing a cabin three-dimensional model under the coordinate system of the image acquisition device based on the plurality of coordinates under the coordinate system of the image acquisition device. According to the method, the high-precision cabin three-dimensional model under the coordinate system of the image acquisition device can be constructed.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional modeling, and in particular to a cockpit three-dimensional modeling method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of science and technology, the application scope of smart cockpits is becoming wider and wider, that is, the intelligentization of car cockpits has become a core element of modern automobile technology. In particular, the visual interaction between the driver and the car is an important direction for the intelligent development of the smart cockpit. Driver visual interaction can not only effectively detect the driver's distraction to improve driving safety, but also achieve a what-you-see-is-what-you-control operation experience through the deep combination of vision and voice, thereby significantly improving the intelligent interaction effect between the driver and the vehicle. For example, by detecting the intersection of the direction vector of the driver's line of sight and each area in the cockpit, it is determined whether the driver's line of sight falls accurately in each area. Based on this, in order to realize the driver's visual interaction, it is necessary to construct a three-dimensional model of the cockpit in the camera coordinate system, and the accuracy of the construction of the three-dimensional model of the cockpit in the camera coordinate system is crucial to the accuracy of the visual interaction.

[0003] At present, the coordinates of multiple areas in the cockpit are determined in the calibration device coordinate system, and then these coordinates are converted into multiple coordinates in the camera coordinate system through a series of complex steps, thereby realizing the construction of the cockpit three-dimensional model in the camera coordinate system. However, the coordinate conversion process of converting the coordinates in the calibration device coordinate system into multiple coordinates in the camera coordinate system is very complicated, and there is a large error in determining the coordinates of multiple areas in the cockpit in the calibration device coordinate system, which leads to a decrease in the accuracy of the cockpit three-dimensional model constructed in the camera coordinate system. Summary of the invention

[0004] The present invention provides a cockpit three-dimensional modeling method, device, equipment, storage medium and program product, which are used to solve the defect of low precision of the cockpit three-dimensional model in the camera coordinate system in the prior art and realize the cockpit three-dimensional model in the image acquisition device coordinate system with high precision.

[0005] The present invention provides a cockpit three-dimensional modeling method, comprising: Determine multiple coordinates in the vehicle coordinate system based on multiple distance data measured by the distance measuring device; any of the distance data is used to represent the distance between a measurement point in the cabin of the target cabin to be modeled and a reference point, the reference point being located in the cabin of the target cabin; Based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, the rotation matrix of the image acquisition device relative to the vehicle coordinate system is determined based on the calibration device and the image acquisition device; Based on a translation vector of the image acquisition device relative to the vehicle coordinate system and a rotation matrix of the image acquisition device relative to the vehicle coordinate system, converting a plurality of coordinates in the vehicle coordinate system into a plurality of coordinates in the image acquisition device coordinate system; Based on the multiple coordinates in the image acquisition device coordinate system, a three-dimensional model of the cabin in the image acquisition device coordinate system is constructed.

[0006] According to a cockpit three-dimensional modeling method provided by the present invention, the calibration device is a checkerboard calibration plate, and the checkerboard calibration plate is arranged in the acquisition direction of the image acquisition device; the rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the following method: Determining internal parameters of the image acquisition device based on a plurality of checkerboard images acquired by the image acquisition device on the checkerboard calibration plate; Based on the internal parameters and the coordinates of the known three-dimensional corner points in the checkerboard calibration plate in the calibration device coordinate system, a rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined.

[0007] According to a cockpit three-dimensional modeling method provided by the present invention, the X-axis of the vehicle coordinate system is parallel to the vehicle's forward direction, and the checkerboard plane of the checkerboard calibration plate is perpendicular to the X-axis of the vehicle coordinate system.

[0008] According to a cockpit three-dimensional modeling method provided by the present invention, the coordinate point of the reference point in the vehicle coordinate system is the origin of the vehicle coordinate system.

[0009] According to a cockpit three-dimensional modeling method provided by the present invention, the rotation matrix of the image acquisition device relative to the calibration device coordinate system, and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, is determined, comprising: Determining the Euler angle of the image acquisition device relative to the calibration device coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system; Based on the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, converting the Euler angle of the image acquisition device relative to the calibration device coordinate system into the Euler angle of the image acquisition device relative to the vehicle coordinate system; Based on the Euler angles of the image acquisition device relative to the vehicle coordinate system, a rotation matrix of the image acquisition device relative to the vehicle coordinate system is determined.

[0010] According to a cockpit three-dimensional modeling method provided by the present invention, the translation vector of the image acquisition device relative to the vehicle coordinate system is determined based on the following method: Determine the coordinates of the image acquisition device in the vehicle coordinate system based on the target distance data measured by the distance measuring device; the target distance data is used to represent the distance between the image acquisition device and the reference point; Based on the coordinates of the image acquisition device in the vehicle coordinate system, a translation vector of the image acquisition device relative to the vehicle coordinate system is determined.

[0011] The present invention also provides a cockpit three-dimensional modeling device, comprising: A coordinate determination module, used to determine a plurality of coordinates in a vehicle coordinate system based on a plurality of distance data measured by a distance measuring device; any of the distance data is used to represent the distance between a measurement point in a cabin of a target cabin to be modeled and a reference point, the reference point being located in the cabin of the target cabin; a matrix determination module, for determining the rotation matrix of the image acquisition device relative to the vehicle coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system; the rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the calibration device and the image acquisition device; A coordinate conversion module, configured to convert a plurality of coordinates in the vehicle coordinate system into a plurality of coordinates in the image acquisition device coordinate system based on a translation vector of the image acquisition device relative to the vehicle coordinate system and a rotation matrix of the image acquisition device relative to the vehicle coordinate system; The three-dimensional modeling module is used to construct a three-dimensional model of the cabin in the image acquisition device coordinate system based on a plurality of coordinates in the image acquisition device coordinate system.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned cockpit three-dimensional modeling methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the three-dimensional modeling method of the cockpit as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned cockpit three-dimensional modeling methods.

[0015] The cockpit three-dimensional modeling method, device, equipment, storage medium and program product provided by the present invention determine multiple coordinates in the vehicle coordinate system based on multiple distance data measured by the distance measuring device, and any distance data is used to represent the distance between the measurement point in the cabin of the target cockpit to be modeled and the reference point, and the reference point is located in the cabin of the target cockpit, so that by setting the reference point, the multiple coordinates in the vehicle coordinate system are cleverly determined, ensuring that the multiple coordinates in the vehicle coordinate system can be provided for subsequent use under any circumstances; based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system, and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, the rotation matrix of the image acquisition device relative to the vehicle coordinate system is quickly determined; based on the translation vector of the image acquisition device relative to the vehicle coordinate system, and the rotation matrix of the image acquisition device relative to the vehicle coordinate system, the multiple coordinates in the vehicle coordinate system are quickly and accurately converted into multiple coordinates in the image acquisition device coordinate system, so as to construct a high-precision cockpit three-dimensional model in the image acquisition device coordinate system based on the multiple coordinates in the image acquisition device coordinate system; in addition, by constructing the cockpit three-dimensional model in the image acquisition device coordinate system in the above manner, no expensive equipment is required, thereby reducing the construction cost of the cockpit three-dimensional model in the image acquisition device coordinate system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of the cockpit three-dimensional modeling method provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the vehicle coordinate system provided by the present invention.

[0019] Figure 3 It is a schematic diagram of the interior of the cockpit provided by the present invention.

[0020] Figure 4 This is the second flow chart of the cockpit three-dimensional modeling method provided by the present invention.

[0021] Figure 5 It is a schematic diagram of the vehicle coordinate system and the calibration plate coordinate system provided by the present invention.

[0022] Figure 6 It is a structural schematic diagram of the cockpit three-dimensional modeling device provided by the present invention.

[0023] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] With the rapid development of science and technology, the application scope of smart cockpits is becoming wider and wider, that is, the intelligence of car cockpits has become a core element of modern automobile technology. In particular, the multimodal interaction between the driver and the car has attracted much attention and is regarded as an important direction for the development of intelligence. Among them, the driver's line of sight interaction, as a key component of multimodal interaction, can not only effectively detect the driver's distraction to improve driving safety, but also achieve the operation experience of what you see is what you control through the deep combination of line of sight and voice, thereby significantly improving the intelligent interaction effect between the driver and the vehicle. For example, by detecting the intersection of the direction vector of the driver's line of sight and each area in the cockpit, it is determined whether the driver's line of sight falls accurately in each area. Based on this, in order to realize the driver's line of sight interaction, it is necessary to construct a three-dimensional model of the cockpit in the camera coordinate system, and the accuracy of the construction of the three-dimensional model of the cockpit in the camera coordinate system is crucial to the accuracy of the line of sight interaction.

[0026] At present, the coordinates of multiple areas in the cabin in the calibration device coordinate system are determined, and then these coordinates are converted into multiple coordinates in the camera coordinate system through a series of complex steps, thereby realizing the construction of the cabin three-dimensional model in the camera coordinate system. However, the coordinate conversion process of converting the coordinates in the calibration device coordinate system into multiple coordinates in the camera coordinate system is very complicated, and there is a large error in determining the coordinates of multiple areas in the cabin in the calibration device coordinate system, which leads to a decrease in the accuracy of the constructed cabin three-dimensional model in the camera coordinate system. In addition, determining the coordinates of multiple areas in the cabin in the calibration device coordinate system requires expensive equipment, resulting in a high cost for constructing the cabin three-dimensional model in the camera coordinate system.

[0027] In addition, the coordinates of each area in the cockpit in the vehicle coordinate system and the installation position and posture of the camera can be obtained from the vehicle digital model file, so as to obtain the coordinates of each area in the camera coordinate system through coordinate system conversion. However, the vehicle digital model file may not be obtained from the car manufacturer, or the vehicle digital model file is lost, resulting in the inability to construct a cockpit 3D model in the camera coordinate system, that is, this method has limitations.

[0028] In view of the above problems, the present invention proposes the following embodiments. Figure 1-Figure 5 The cockpit three-dimensional modeling method of the present invention is described.

[0029] Figure 1 It is one of the flow charts of the cockpit three-dimensional modeling method provided by the present invention, such as Figure 1 As shown, the cockpit three-dimensional modeling method includes the following steps 110, 120, 130 and 140.

[0030] Step 110 : determining a plurality of coordinates in a vehicle coordinate system based on a plurality of distance data measured by a distance measuring device.

[0031] Here, the distance measuring device is used to measure the distance between the measurement point in the target cabin and the reference point. In one embodiment, the distance measuring device can be a laser rangefinder.

[0032] Among them, any of the distance data is used to represent the distance between a measurement point in the cabin of the target cabin to be modeled and a reference point, and the reference point is located in the cabin of the target cabin.

[0033] Furthermore, in order to reduce the amount of calculation, the distance measuring device only needs to measure the distance between the key point in the target cabin and the reference point, that is, any distance data is used to represent the distance between the key point in the target cabin and the reference point. Figure 2 and Figure 3 As shown, Figure 2 is a schematic diagram of a vehicle coordinate system provided by the present invention, Figure 2 The X-axis of the vehicle coordinate system is parallel to the vehicle's forward direction, the Z-axis of the vehicle coordinate system is parallel to the vehicle's height direction, and the Y-axis of the vehicle coordinate system is parallel to the vehicle's width direction. Figure 3 is a schematic diagram of the interior of the cockpit provided by the present invention, Figure 3 The X-axis of the vehicle coordinate system is parallel to the forward direction of the vehicle, the Z-axis of the vehicle coordinate system is parallel to the height direction of the vehicle, the Y-axis of the vehicle coordinate system is parallel to the width direction of the vehicle, and the coordinates of the reference point in the vehicle coordinate system are the origin coordinates of the vehicle coordinate system; assuming that when measuring the front windshield area in the target cockpit, the four corner points of the front windshield (upper left corner point 1, upper right corner point 2, lower left corner point 3, lower right corner point 4) are selected as key points, so that in the front windshield area, only the distances between these four key points and the reference point need to be measured.

[0034] It should be noted that a reference point is pre-selected in the cabin of the target cockpit, so that the coordinates of the reference point in the vehicle coordinate system are known, which facilitates accurate determination of the coordinates of the corresponding measurement point in the cabin in the vehicle coordinate system based on the distance data measured by the ranging device.

[0035] In one embodiment, if Figure 2 and Figure 3 As shown, the X-axis of the vehicle coordinate system is parallel to the forward direction of the vehicle, the Z-axis of the vehicle coordinate system is parallel to the height direction of the vehicle, and the coordinate point of the reference point in the vehicle coordinate system is located on the XZ plane of the vehicle coordinate system. Considering that the vehicle is almost symmetrical on the left and right, the reference point is selected to be located on the XZ plane of the vehicle coordinate system, which makes it easier to more accurately determine the coordinates of the corresponding measurement point in the cabin in the vehicle coordinate system based on the distance data measured by the ranging device.

[0036] Specifically, based on a plurality of distance data measured by the distance measuring device, coordinates of a plurality of in-cabin measurement points in the vehicle coordinate system are determined.

[0037] Furthermore, if there are multiple image acquisition devices in the smart cockpit, the three-dimensional cockpit model in the coordinate system of each image acquisition device can be constructed respectively. And the three-dimensional cockpit model in the coordinate system of each image acquisition device can be a three-dimensional model of different areas of the vehicle cockpit, that is, based on the multiple distance data measured by the ranging device, the multiple coordinates of the target area in the vehicle coordinate system are determined, and any distance data is used to represent the distance between the cabin measurement point and the reference point in the target area of ​​the target cockpit.

[0038] In one embodiment, the vehicle coordinate system is a coordinate system that complies with the ISO 4130:1978 standard vehicle coordinate system.

[0039] Step 120 , based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, determine the rotation matrix of the image acquisition device relative to the vehicle coordinate system.

[0040] The rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the calibration device and the image acquisition device. That is, the external parameters (including the rotation matrix) of the image acquisition device relative to the calibration device coordinate system can be obtained by calibrating the image acquisition device through the calibration device.

[0041] Here, the image acquisition device may be a camera. The calibration device coordinate system is a coordinate system of the calibration device. Further, the calibration device may be a checkerboard calibration plate, so as to more accurately determine the rotation matrix of the image acquisition device relative to the calibration device coordinate system.

[0042] It should be noted that since the vehicle coordinate system and the calibration device coordinate system are both known coordinate systems, based on this, the conversion relationship between the vehicle coordinate system and the calibration device coordinate system can be predetermined. Figure 2 and Figure 3 As shown, if the X-axis of the vehicle coordinate system is parallel to the vehicle's forward direction and the plane of the calibration device is perpendicular to the X-axis of the vehicle coordinate system, it is easy to quickly and accurately determine the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, thereby improving the accuracy of determining the rotation matrix of the image acquisition device relative to the vehicle coordinate system, and ultimately improving the construction accuracy of the cockpit three-dimensional model in the image acquisition device coordinate system.

[0043] Specifically, based on the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, the rotation matrix of the image acquisition device relative to the calibration device coordinate system is converted into the rotation matrix of the image acquisition device relative to the vehicle coordinate system.

[0044] Step 130 , based on the translation vector of the image acquisition device relative to the vehicle coordinate system and the rotation matrix of the image acquisition device relative to the vehicle coordinate system, convert a plurality of coordinates in the vehicle coordinate system into a plurality of coordinates in the image acquisition device coordinate system.

[0045] Here, the image acquisition device coordinate system is the coordinate system of the image acquisition device, such as a camera coordinate system.

[0046] Exemplarily, the conversion formula for converting multiple coordinates in the vehicle coordinate system into multiple coordinates in the image acquisition device coordinate system is as follows: ; In the formula, represents the coordinates in the image acquisition device coordinate system, represents the rotation matrix of the image acquisition device relative to the vehicle coordinate system, represents the coordinates in the vehicle coordinate system, Represents the translation vector of the image acquisition device relative to the vehicle coordinate system.

[0047] Step 140: construct a three-dimensional model of the cabin in the image acquisition device coordinate system based on the multiple coordinates in the image acquisition device coordinate system.

[0048] It should be understood that based on the multiple coordinates in the image acquisition device coordinate system, the three-dimensional model of the cockpit in the image acquisition device coordinate system can be reconstructed.

[0049] The cockpit three-dimensional modeling method provided by the embodiment of the present invention determines multiple coordinates in the vehicle coordinate system based on multiple distance data measured by the distance measuring device, and any distance data is used to represent the distance between the measurement point in the cabin of the target cockpit to be modeled and the reference point, and the reference point is located in the cabin of the target cockpit, so that the multiple coordinates in the vehicle coordinate system are cleverly determined by setting the reference point, ensuring that the multiple coordinates in the vehicle coordinate system can be provided for subsequent use under any circumstances; based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system, and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, the rotation matrix of the image acquisition device relative to the vehicle coordinate system is quickly determined; based on the translation vector of the image acquisition device relative to the vehicle coordinate system, and the rotation matrix of the image acquisition device relative to the vehicle coordinate system, the multiple coordinates in the vehicle coordinate system are quickly and accurately converted into multiple coordinates in the image acquisition device coordinate system, so as to construct a high-precision cockpit three-dimensional model in the image acquisition device coordinate system based on the multiple coordinates in the image acquisition device coordinate system; in addition, by constructing the cockpit three-dimensional model in the image acquisition device coordinate system in the above manner, no expensive equipment is required, thereby reducing the construction cost of the cockpit three-dimensional model in the image acquisition device coordinate system.

[0050] Based on any of the above embodiments, in the method, the calibration device is a checkerboard calibration plate, and the checkerboard calibration plate is arranged in the acquisition direction of the image acquisition device.

[0051] Here, the checkerboard calibration plate is composed of black and white square grids, forming a checkerboard pattern. These grids have regular sizes and shapes, so that their intersections become easy-to-detect feature points. Based on this, the accuracy of determining the rotation matrix of the image acquisition device relative to the calibration device coordinate system is improved.

[0052] Since the image acquisition device needs to acquire the checkerboard image of the checkerboard calibration plate, the checkerboard calibration plate needs to be arranged in the acquisition direction of the image acquisition device. Further, the checkerboard calibration plate is arranged right in front of the image acquisition device.

[0053] Accordingly, the rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the following method: Determining internal parameters of the image acquisition device based on a plurality of checkerboard images acquired by the image acquisition device on the checkerboard calibration plate; Based on the internal parameters and the coordinates of the known three-dimensional corner points in the checkerboard calibration plate in the calibration device coordinate system, a rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined.

[0054] Here, the checkerboard image is an image corresponding to the checkerboard pattern on the checkerboard calibration plate.

[0055] Specifically, the camera calibration is performed based on a plurality of checkerboard images to obtain the internal parameters of the image acquisition device. Further, the camera calibration is performed based on a plurality of checkerboard images acquired by the image acquisition device at a plurality of angles of the checkerboard calibration plate. That is, the plurality of checkerboard images include checkerboard images at a plurality of shooting angles, thereby improving the accuracy of determining the internal parameters of the image acquisition device, and further improving the accuracy of determining the rotation matrix of the image acquisition device relative to the coordinate system of the calibration device.

[0056] For example, based on a plurality of checkerboard images, calibration software is used to calculate internal parameters of the image acquisition device, which may include focal length, principal point, and distortion parameters (such as radial distortion and tangential distortion).

[0057] Specifically, based on the internal parameters of the image acquisition device, the two-dimensional corner points detected in the checkerboard image are matched with the known three-dimensional corner points in the checkerboard calibration plate, so as to determine the rotation matrix of the image acquisition device relative to the calibration device coordinate system based on the matching results. That is, by comparing the 2D corner points obtained by projection with the 2D corner points actually detected on the image, the corresponding relationship between them is found to determine the rotation matrix of the image acquisition device relative to the calibration device coordinate system. Furthermore, based on the internal parameters and the coordinates of the known three-dimensional corner points in the checkerboard calibration plate in the calibration device coordinate system, the translation vector of the image acquisition device relative to the calibration device coordinate system can also be determined, so as to determine the external parameters of the image acquisition device relative to the calibration device coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system and the translation vector of the image acquisition device relative to the calibration device coordinate system.

[0058] The cockpit three-dimensional modeling method provided by the embodiment of the present invention accurately determines the internal parameters of the image acquisition device based on multiple checkerboard images acquired by the image acquisition device on the checkerboard calibration plate, and the calibration device is a checkerboard calibration plate, and the checkerboard calibration plate is arranged in the acquisition direction of the image acquisition device, thereby improving the accuracy of determining the internal parameters of the image acquisition device, and then improving the accuracy of determining the rotation matrix of the image acquisition device relative to the calibration device coordinate system, and finally improving the construction accuracy of the cockpit three-dimensional model in the image acquisition device coordinate system; based on the internal parameters and the coordinates of the known three-dimensional corner points in the checkerboard calibration plate in the calibration device coordinate system, the rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined. Since it is based on accurate internal parameters and known three-dimensional corner points, the rotation matrix of the image acquisition device relative to the calibration device coordinate system is accurately determined, and finally the construction accuracy of the cockpit three-dimensional model in the image acquisition device coordinate system is improved.

[0059] Based on any of the above embodiments, in the method, if Figure 2 and Figure 3As shown, the X-axis of the vehicle coordinate system is parallel to the forward direction of the vehicle, and the checkerboard plane of the checkerboard calibration plate is perpendicular to the X-axis of the vehicle coordinate system.

[0060] Here, the checkerboard plane is a calibration plate plane, or a plane with a checkerboard pattern. Further, the checkerboard plane is located on the XY plane of the calibration device coordinate system, that is, the checkerboard plane is perpendicular to the Z axis of the calibration device coordinate system.

[0061] The cockpit three-dimensional modeling method provided by the embodiment of the present invention, if the X-axis of the vehicle coordinate system is parallel to the vehicle's forward direction and the plane of the checkerboard calibration plate is perpendicular to the X-axis of the vehicle coordinate system, it is convenient to quickly and accurately determine the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, thereby improving the accuracy of determining the rotation matrix of the image acquisition device relative to the vehicle coordinate system, and ultimately improving the construction accuracy of the cockpit three-dimensional model in the image acquisition device coordinate system.

[0062] Based on any of the above embodiments, Figure 3 As shown, in this method, the coordinate point of the reference point in the vehicle coordinate system is the origin of the vehicle coordinate system, that is, the vehicle coordinate system is constructed based on the reference point.

[0063] Furthermore, if Figure 2 and Figure 3 As shown, the X-axis of the vehicle coordinate system is parallel to the forward direction of the vehicle, the Z-axis of the vehicle coordinate system is parallel to the height direction of the vehicle, and the coordinate point of the reference point in the vehicle coordinate system is located on the XZ plane of the vehicle coordinate system. Considering that the left and right sides of the vehicle are almost symmetrical, the reference point is selected to be located on the XZ plane of the vehicle coordinate system, which makes it easier to more accurately determine the coordinates of the corresponding measurement points in the cabin in the vehicle coordinate system based on the distance data measured by the ranging device, thereby ultimately improving the construction accuracy of the three-dimensional model of the cabin in the coordinate system of the image acquisition device.

[0064] In the cockpit three-dimensional modeling method provided by an embodiment of the present invention, the coordinate point of the reference point in the vehicle coordinate system is the origin of the vehicle coordinate system, so there is no need for conversion. Based on multiple distance data measured by the ranging device, multiple coordinates in the vehicle coordinate system can be quickly determined, thereby improving the construction efficiency of the cockpit three-dimensional model in the image acquisition device coordinate system.

[0065] Based on any of the above embodiments, Figure 4 FIG. 2 is a flow chart of the cockpit three-dimensional modeling method provided by the present invention. Figure 4 As shown, the above step 120 includes: step 121, step 122 and step 123.

[0066] Step 121: Determine the Euler angle of the image acquisition device relative to the calibration device coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system.

[0067] Here, the Euler angle includes the yaw angle (Yaw), the pitch angle (Pitch), and the roll angle (Roll).

[0068] For example, Figure 5 As shown in the figure, the yaw angle represents the image acquisition device around the calibration plate coordinate system (calibration device coordinate system). The pitch angle refers to the rotation angle of the image acquisition device around the calibration plate coordinate system (calibration device coordinate system). The roll angle refers to the rotation angle of the image acquisition device around the calibration plate coordinate system (calibration device coordinate system). The angle of axis rotation.

[0069] Step 122: based on the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, convert the Euler angles of the image acquisition device relative to the calibration device coordinate system into the Euler angles of the image acquisition device relative to the vehicle coordinate system.

[0070] Step 123: Determine a rotation matrix of the image acquisition device relative to the vehicle coordinate system based on the Euler angles of the image acquisition device relative to the vehicle coordinate system.

[0071] For example, Figure 5 As shown, the calibration device is a checkerboard calibration plate, the X-axis of the vehicle coordinate system is parallel to the vehicle's forward direction, the checkerboard plane of the checkerboard calibration plate is perpendicular to the X-axis of the vehicle coordinate system, and the checkerboard plane is perpendicular to the calibration plate coordinate system (calibration device coordinate system). Axis; The yaw angle of the image acquisition device relative to the calibration device coordinate system represents the yaw angle of the image acquisition device around the calibration plate coordinate system (calibration device coordinate system). The pitch angle of the image acquisition device relative to the calibration device coordinate system represents the rotation angle of the image acquisition device around the calibration board coordinate system (calibration device coordinate system). The roll angle of the image acquisition device relative to the calibration device coordinate system represents the rotation angle of the image acquisition device around the calibration plate coordinate system (calibration device coordinate system). The angle of axis rotation; based on this, the calculation formula of the rotation matrix of the image acquisition device relative to the vehicle coordinate system is as follows: ; Where R represents the rotation matrix of the image acquisition device relative to the vehicle coordinate system; Represents the pitch angle of the image acquisition device relative to the calibration device coordinate system, reference Figure 5 , you can As the rotation angle of the image acquisition device around the Z axis of the vehicle coordinate system, A rotation matrix representing the angle of rotation of the image acquisition device around the Z axis of the vehicle coordinate system; Represents the roll angle of the image acquisition device relative to the calibration device coordinate system, reference Figure 5 , you can As the rotation angle of the image acquisition device around the Y axis of the vehicle coordinate system, A rotation matrix representing the angle of rotation of the image acquisition device around the Y axis of the vehicle coordinate system; Represents the yaw angle of the image acquisition device relative to the calibration device coordinate system, reference Figure 5 , you can As the rotation angle of the image acquisition device around the X-axis of the vehicle coordinate system, The rotation matrix represents the angle of rotation of the image acquisition device around the X-axis of the vehicle coordinate system.

[0072] The cockpit three-dimensional modeling method provided by the embodiment of the present invention determines the Euler angles of the image acquisition device relative to the calibration device coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system, converts the Euler angles of the image acquisition device relative to the calibration device coordinate system into the Euler angles of the image acquisition device relative to the vehicle coordinate system based on the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, and determines the rotation matrix of the image acquisition device relative to the vehicle coordinate system based on the Euler angles of the image acquisition device relative to the vehicle coordinate system, thereby improving the accuracy of determining the rotation matrix of the image acquisition device relative to the vehicle coordinate system, and ultimately improving the construction accuracy of the cockpit three-dimensional model in the image acquisition device coordinate system.

[0073] Based on any of the above embodiments, in the method, the translation vector of the image acquisition device relative to the vehicle coordinate system is determined based on the following method: Determine the coordinates of the image acquisition device in the vehicle coordinate system based on the target distance data measured by the distance measuring device; the target distance data is used to represent the distance between the image acquisition device and the reference point; Based on the coordinates of the image acquisition device in the vehicle coordinate system, a translation vector of the image acquisition device relative to the vehicle coordinate system is determined.

[0074] It should be noted that since the target distance data is used to represent the distance between the image acquisition device and the reference point, the coordinates of the image acquisition device in the vehicle coordinate system can be accurately determined based on the target distance data, and then the translation vector of the image acquisition device relative to the vehicle coordinate system can be accurately determined.

[0075] The cockpit three-dimensional modeling method provided by the embodiment of the present invention determines the coordinates of the image acquisition device in the vehicle coordinate system based on the target distance data measured by the distance measuring device, and the target distance data is used to represent the distance between the image acquisition device and a reference point, and the reference point is located inside the cabin of the target cockpit. Therefore, by setting the reference point, the coordinates of the image acquisition device in the vehicle coordinate system are cleverly determined, ensuring that the translation vector of the image acquisition device relative to the vehicle coordinate system can be provided for the subsequent operation under any circumstances; at the same time, the above method can accurately determine the translation vector of the image acquisition device relative to the vehicle coordinate system, and finally improve the construction accuracy of the cockpit three-dimensional model in the image acquisition device coordinate system.

[0076] Based on the above embodiments, the present invention provides a convenient and easy-to-implement cockpit 3D modeling method. By using conventional distance measuring tools and calibration plate devices, a high-precision cockpit 3D model in the image acquisition device coordinate system can be constructed.

[0077] The cockpit three-dimensional modeling device provided by the present invention is described below. The cockpit three-dimensional modeling device described below and the cockpit three-dimensional modeling method described above can be referred to each other.

[0078] Figure 6 Schematic diagram of the structure of the cockpit three-dimensional modeling device provided by the present invention. Figure 6 As shown, the cockpit three-dimensional modeling device includes a coordinate determination module 610, a matrix determination module 620, a coordinate conversion module 630 and a three-dimensional modeling module 640.

[0079] The coordinate determination module 610 is used to determine multiple coordinates in the vehicle coordinate system based on multiple distance data measured by the ranging device; any of the distance data is used to represent the distance between a measurement point inside the cabin of the target cabin to be modeled and a reference point, and the reference point is located inside the cabin of the target cabin.

[0080] The matrix determination module 620 is used to determine the rotation matrix of the image acquisition device relative to the vehicle coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system and the transformation relationship between the vehicle coordinate system and the calibration device coordinate system; the rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the calibration device and the image acquisition device.

[0081] The coordinate conversion module 630 is used to convert multiple coordinates in the vehicle coordinate system into multiple coordinates in the image acquisition device coordinate system based on the translation vector of the image acquisition device relative to the vehicle coordinate system and the rotation matrix of the image acquisition device relative to the vehicle coordinate system.

[0082] The three-dimensional modeling module 640 is used to construct a three-dimensional model of the cabin in the image acquisition device coordinate system based on a plurality of coordinates in the image acquisition device coordinate system.

[0083] The cockpit three-dimensional modeling device provided by the embodiment of the present invention determines multiple coordinates in the vehicle coordinate system based on multiple distance data measured by the distance measuring device, and any distance data is used to represent the distance between the measurement point in the cabin of the target cockpit to be modeled and the reference point, and the reference point is located in the cabin of the target cockpit, so that the multiple coordinates in the vehicle coordinate system are cleverly determined by setting the reference point, ensuring that the multiple coordinates in the vehicle coordinate system can be provided for subsequent use under any circumstances; based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system, and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, the rotation matrix of the image acquisition device relative to the vehicle coordinate system is quickly determined, and based on the translation vector of the image acquisition device relative to the vehicle coordinate system, and the rotation matrix of the image acquisition device relative to the vehicle coordinate system, the multiple coordinates in the vehicle coordinate system are quickly and accurately converted into multiple coordinates in the image acquisition device coordinate system, so as to construct a high-precision cockpit three-dimensional model in the image acquisition device coordinate system based on the multiple coordinates in the image acquisition device coordinate system; in addition, by constructing the cockpit three-dimensional model in the image acquisition device coordinate system in the above manner, no expensive equipment is required, thereby reducing the construction cost of the cockpit three-dimensional model in the image acquisition device coordinate system.

[0084] Based on any of the above embodiments, the calibration device is a checkerboard calibration plate, and the checkerboard calibration plate is arranged in the acquisition direction of the image acquisition device; the matrix determination module 620 is also used for: Determining internal parameters of the image acquisition device based on a plurality of checkerboard images acquired by the image acquisition device on the checkerboard calibration plate; Based on the internal parameters and the coordinates of the known three-dimensional corner points in the checkerboard calibration plate in the calibration device coordinate system, a rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined.

[0085] Based on any of the above embodiments, the X-axis of the vehicle coordinate system is parallel to the forward direction of the vehicle, and the checkerboard plane of the checkerboard calibration plate is perpendicular to the X-axis of the vehicle coordinate system.

[0086] Based on any of the above embodiments, the coordinate point of the reference point in the vehicle coordinate system is the origin of the vehicle coordinate system.

[0087] Based on any of the above embodiments, the matrix determination module 620 is specifically used for: Determining the Euler angle of the image acquisition device relative to the calibration device coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system; Based on the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, converting the Euler angle of the image acquisition device relative to the calibration device coordinate system into the Euler angle of the image acquisition device relative to the vehicle coordinate system; Based on the Euler angles of the image acquisition device relative to the vehicle coordinate system, a rotation matrix of the image acquisition device relative to the vehicle coordinate system is determined.

[0088] Based on any of the above embodiments, the coordinate conversion module 630 is further used for: Determine the coordinates of the image acquisition device in the vehicle coordinate system based on the target distance data measured by the distance measuring device; the target distance data is used to represent the distance between the image acquisition device and the reference point; Based on the coordinates of the image acquisition device in the vehicle coordinate system, a translation vector of the image acquisition device relative to the vehicle coordinate system is determined.

[0089] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor (processor) 710 , a communication interface (Communications Interface) 720 , a memory (memory) 730 and a communication bus 740 , wherein the processor 710 , the communication interface 720 , and the memory 730 communicate with each other through the communication bus 740 . The processor 710 can call the logic instructions in the memory 730 to execute the cockpit three-dimensional modeling method, which includes: determining multiple coordinates in the vehicle coordinate system based on multiple distance data measured by the ranging device; any of the distance data is used to represent the distance between the measurement point in the cabin of the target cockpit to be modeled and the reference point, and the reference point is located in the cabin of the target cockpit; based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system, and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, determining the rotation matrix of the image acquisition device relative to the vehicle coordinate system; the rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the calibration device and the image acquisition device; based on the translation vector of the image acquisition device relative to the vehicle coordinate system, and the rotation matrix of the image acquisition device relative to the vehicle coordinate system, converting the multiple coordinates in the vehicle coordinate system into the multiple coordinates in the image acquisition device coordinate system; based on the multiple coordinates in the image acquisition device coordinate system, constructing a cockpit three-dimensional model in the image acquisition device coordinate system.

[0090] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0091] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cockpit three-dimensional modeling method provided by the above methods, the method comprising: determining multiple coordinates in a vehicle coordinate system based on multiple distance data measured by a ranging device; any of the distance data is used to represent the distance between a measurement point in the cabin of a target cockpit to be modeled and a reference point, and the reference point is located in the cabin of the target cockpit; based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system, and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, determining the rotation matrix of the image acquisition device relative to the vehicle coordinate system; the rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the calibration device and the image acquisition device; based on the translation vector of the image acquisition device relative to the vehicle coordinate system, and the rotation matrix of the image acquisition device relative to the vehicle coordinate system, converting the multiple coordinates in the vehicle coordinate system into the multiple coordinates in the image acquisition device coordinate system; based on the multiple coordinates in the image acquisition device coordinate system, constructing a cockpit three-dimensional model in the image acquisition device coordinate system.

[0092] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the cockpit three-dimensional modeling method provided by the above methods when the computer program is executed, the method comprising: determining a plurality of coordinates in a vehicle coordinate system based on a plurality of distance data measured by a distance measuring device; any of the distance data is used to represent the distance between a measurement point in the cabin of a target cockpit to be modeled and a reference point, the reference point being located in the cabin of the target cockpit; determining a rotation matrix of the image acquisition device relative to the vehicle coordinate system based on a rotation matrix of the image acquisition device relative to the calibration device coordinate system, and a conversion relationship between the vehicle coordinate system and the calibration device coordinate system; the rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the calibration device and the image acquisition device; converting a plurality of coordinates in the vehicle coordinate system into a plurality of coordinates in the image acquisition device coordinate system based on a translation vector of the image acquisition device relative to the vehicle coordinate system, and a rotation matrix of the image acquisition device relative to the vehicle coordinate system; and constructing a cockpit three-dimensional model in the image acquisition device coordinate system based on the plurality of coordinates in the image acquisition device coordinate system.

[0093] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cockpit three-dimensional modeling method, characterized in that: include: Determine multiple coordinates in the vehicle coordinate system based on multiple distance data measured by the distance measuring device; any of the distance data is used to represent the distance between a measurement point in the cabin of the target cabin to be modeled and a reference point, the reference point being located in the cabin of the target cabin; Determine the rotation matrix of the image acquisition device relative to the vehicle coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system; The rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the calibration device and the image acquisition device; Based on a translation vector of the image acquisition device relative to the vehicle coordinate system and a rotation matrix of the image acquisition device relative to the vehicle coordinate system, converting a plurality of coordinates in the vehicle coordinate system into a plurality of coordinates in the image acquisition device coordinate system; Based on the multiple coordinates in the image acquisition device coordinate system, a three-dimensional model of the cabin in the image acquisition device coordinate system is constructed.

2. The cockpit three-dimensional modeling method according to claim 1, characterized in that: The calibration device is a checkerboard calibration plate, and the checkerboard calibration plate is arranged in the acquisition direction of the image acquisition device; the rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the following method: Determining internal parameters of the image acquisition device based on a plurality of checkerboard images acquired by the image acquisition device on the checkerboard calibration plate; Based on the internal parameters and the coordinates of the known three-dimensional corner points in the checkerboard calibration plate in the calibration device coordinate system, a rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined.

3. The cockpit three-dimensional modeling method according to claim 2, characterized in that: The X-axis of the vehicle coordinate system is parallel to the forward direction of the vehicle, and the checkerboard plane of the checkerboard calibration plate is perpendicular to the X-axis of the vehicle coordinate system.

4. The cockpit three-dimensional modeling method according to claim 1, characterized in that: The coordinate point of the reference point in the vehicle coordinate system is the origin of the vehicle coordinate system.

5. The cockpit three-dimensional modeling method according to any one of claims 1 to 4, characterized in that: The determining of the rotation matrix of the image acquisition device relative to the vehicle coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system and the conversion relationship between the vehicle coordinate system and the calibration device coordinate system includes: Determining the Euler angle of the image acquisition device relative to the calibration device coordinate system based on the rotation matrix of the image acquisition device relative to the calibration device coordinate system; Based on the conversion relationship between the vehicle coordinate system and the calibration device coordinate system, converting the Euler angle of the image acquisition device relative to the calibration device coordinate system into the Euler angle of the image acquisition device relative to the vehicle coordinate system; Based on the Euler angles of the image acquisition device relative to the vehicle coordinate system, a rotation matrix of the image acquisition device relative to the vehicle coordinate system is determined.

6. The cockpit three-dimensional modeling method according to any one of claims 1 to 4, characterized in that: The translation vector of the image acquisition device relative to the vehicle coordinate system is determined based on the following method: Determine the coordinates of the image acquisition device in the vehicle coordinate system based on the target distance data measured by the distance measuring device; the target distance data is used to represent the distance between the image acquisition device and the reference point; Based on the coordinates of the image acquisition device in the vehicle coordinate system, a translation vector of the image acquisition device relative to the vehicle coordinate system is determined.

7. A cockpit three-dimensional modeling device, characterized in that: include: A coordinate determination module, used to determine a plurality of coordinates in a vehicle coordinate system based on a plurality of distance data measured by a distance measuring device; any of the distance data is used to represent the distance between a measurement point in a cabin of a target cabin to be modeled and a reference point, the reference point being located in the cabin of the target cabin; A matrix determination module, configured to determine a rotation matrix of the image acquisition device relative to the vehicle coordinate system based on a rotation matrix of the image acquisition device relative to the calibration device coordinate system and a conversion relationship between the vehicle coordinate system and the calibration device coordinate system; The rotation matrix of the image acquisition device relative to the calibration device coordinate system is determined based on the calibration device and the image acquisition device; A coordinate conversion module, configured to convert a plurality of coordinates in the vehicle coordinate system into a plurality of coordinates in the image acquisition device coordinate system based on a translation vector of the image acquisition device relative to the vehicle coordinate system and a rotation matrix of the image acquisition device relative to the vehicle coordinate system; The three-dimensional modeling module is used to construct a three-dimensional model of the cabin in the image acquisition device coordinate system based on a plurality of coordinates in the image acquisition device coordinate system.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the cockpit three-dimensional modeling method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cockpit three-dimensional modeling method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the cockpit three-dimensional modeling method according to any one of claims 1 to 6 is implemented.

Images