360-degree surround stitching method for tractor trailer
By setting the IMU on the traction trailer to measure the angular velocity difference in real time, calculate the relative rotation angle, and dynamically adjust the image stitching algorithm, the problem of adjusting the image stitching algorithm of the traction trailer is solved, and the accuracy and stability of 360-degree surround splicing is improved.
Patent Information
- Application Number
- CN202111136237.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-27
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-09-27
AI Technical Summary
In the field of 360-degree surround view splicing, the positional relationship between the camera and the vehicle is changed due to the change in the rotation angle between the front and the trailer. The existing technology is difficult to dynamically adjust the image splicing algorithm and cannot accurately splice the images. .
By setting the first IMU and the second IMU on the traction vehicle front and trailer, the angular velocity difference is measured in real time to calculate the relative rotation angle, and dynamically adjust the image stitching algorithm to realize 360-degree surround splicing of front, rear, left and right view images.
Real-time high-frequency output angle information, dynamically adjust the image splicing algorithm, improve the accuracy of 360-degree surround splicing, and lossless transformation, low cost and wide application range.
Smart Images

Figure CN113850723B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a 360-degree surround-view splicing method for a tractor trailer. Background Art
[0002] In the field of 360-degree surround view stitching, images from the front, back, left and right cameras are usually stitched together in a passenger car. At present, a general passenger car can be roughly regarded as a rigid body. The positional relationship between the cameras and between the camera and the vehicle will not change during the operation of the car, so the image stitching algorithm is fixed. However, it is different for trailers. A rotation angle will be generated between the front of the car and the trailer, thereby changing the positional relationship between the cameras and between the camera and the vehicle. Therefore, the image stitching algorithm must be dynamically adjusted according to the change in this angle. However, how to measure and estimate the angle between the front of the car and the trailer is a technical problem, which makes it impossible to dynamically adjust the image stitching algorithm according to the change in this angle. Summary of the invention
[0003] The present invention provides a 360-degree surround view splicing method for a tractor trailer, which can effectively solve the above problems.
[0004] The present invention is achieved in that:
[0005] A 360-degree surround view stitching method for a tractor trailer, the tractor trailer comprising a tractor head and a trailer; a first IMU is further arranged on the tractor head, and a second IMU is further arranged on the trailer to communicate with the first IMU; the first IMU and the second IMU are arranged on a straight line parallel to the length direction of the vehicle; and a camera is used to obtain the front, rear, left and right views of the vehicle, the method comprising the following steps:
[0006] S1, obtaining the angular velocity of the trailer measured by the second IMU and recorded as av2, obtaining the angular velocity of the tractor head measured by the first IMU and recorded as av1, and obtaining the front, rear, left and right views of the vehicle in real time;
[0007] S2, obtaining the angular velocity difference between the tractor head and the trailer, diff_av=(av1-av2);
[0008] S3, obtaining the relative rotation angle delt_a between the tractor head and the trailer at the current moment through the formula: delt_a=diff_av / f;
[0009] S4, accumulating the relative rotation angle delt_a of each frame, and then obtaining the angle a=Σdelt_a between the tractor head and the trailer;
[0010] S5, dynamically adjusting the entire image stitching algorithm according to the change of the angle a, and then performing 360-degree surround stitching on the acquired front, rear, left, and right view images.
[0011] The beneficial effects of the present invention are as follows: the 360-degree surround view stitching method for a tractor trailer of the present invention can output the turning angle information in real time and at high frequency, so that the entire image stitching algorithm can be dynamically adjusted according to the change of the angle a, and then the acquired front, rear, left and right view images are stitched in 360 degrees, thereby improving the accuracy of the 360-degree surround view stitching; in addition, the present invention is non-destructive to the vehicle, has a small modification workload, low cost, high stability and a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0013] Figure 1 It is a schematic structural diagram of a trailer with dual IMUs provided in an embodiment of the present invention.
[0014] Figure 2 It is a flow chart of a method for real-time acquisition of trailer angle in a trailer with dual IMUs provided in an embodiment of the present invention.
[0015] Figure 3 It is a flow chart of a 360-degree surround view stitching method for a tractor trailer in a trailer with dual IMUs provided in an embodiment of the present invention.
[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0017] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0018] Reference Figure 1 As shown, an embodiment of the present invention provides a tractor trailer with dual IMU inertial sensors, which includes a tractor head 1 and a trailer 2. The embodiment of the present invention takes a trailer as an example, but is not limited to a trailer and is also applicable to a semi-trailer.
[0019] A first IMU11 is further provided on the tractor head 1, and a second IMU21 is further provided on the trailer 2 to be communicated with and connected to the first IMU11. The first IMU11 and the second IMU21 are provided on a straight line parallel to the length direction of the vehicle. The X-axis of the first IMU11 faces forward, the Y-axis faces left, and the Z-axis faces upward, and is kept horizontally arranged; the X-axis of the second IMU21 faces forward, the Y-axis faces left, and the Z-axis faces upward, and is kept horizontally arranged. Preferably, the first IMU11 and the second IMU21 are provided on the central axis along the length direction of the vehicle. The first IMU11 can be provided at the top, middle, or bottom of the tractor head 1. In one embodiment, the first IMU11 is provided at the top of the tractor head 1. The second IMU21 can be provided at the top, middle, or bottom of the trailer 2. In one embodiment, the second IMU21 is provided at the top of the trailer 2. Furthermore, since the trailer 2 is relatively long, in order to more accurately obtain the angle between the tractor head 1 and the trailer 2, the second IMU 21 is more preferably disposed at the middle and rear portion of the trailer 2. In another embodiment, the second IMU 21 is disposed at the rear portion of the top of the trailer 2.
[0020] When the vehicle starts, the tractor head 1 and the trailer 2 move to any angle, and the two IMUs simultaneously collect real-time data at a high frequency f. The second IMU 21 on the trailer 2 sends the collected data to the first IMU 11 through the CAN network / serial port, and the first IMU 11 also collects data. At this time, the angle between the tractor head 1 and the trailer 2 can be calculated by the first IMU 11. The specific calculation is as follows:
[0021] The angular velocity of the trailer 2 measured by the second IMU 21 is obtained and recorded as av2, and the angular velocity of the tractor head 1 measured by the first IMU 11 is obtained and recorded as av1; the difference in angular velocity between the tractor head 1 and the trailer 2 is obtained as diff_av=(av1-av2); further, the relative rotation angle delt_a between the tractor head 1 and the trailer 2 at the current moment is obtained by the formula: delt_a=diff_av / f; finally, the relative rotation angle delt_a of each frame is accumulated to obtain the angle a=Σdelt_a between the tractor head 1 and the trailer 2.
[0022] As a further improvement, the tractor trailer with dual IMU inertial sensors further includes a plurality of cameras arranged on the tractor head 1 and the trailer 2. Specifically, the tractor head 1 includes a first camera 12 arranged on both sides of the front of the tractor head 1; the trailer 2 includes a second camera 21 arranged on both sides of the rear of the trailer 2. The types of the first camera 12 and the second camera 21 are not limited. In one embodiment, the first camera 12 and the second camera 21 are both fisheye cameras. Through the dynamic change of the real-time angle a between the tractor head 1 and the trailer 2, the front, rear, left and right view images obtained by the first camera 12 and the second camera 21 can be spliced in a 360-degree surround view, so that the entire image stitching algorithm can be dynamically adjusted according to the change of the real-time angle a.
[0023] As a further improvement, the tractor trailer with dual IMU further includes a processor which is not shown in the figure. The processor is further used to obtain the angle a between the tractor head 1 and the trailer 2, and the front, rear, left and right views of the first camera 12 and the second camera 21; and is further used to dynamically adjust the entire image stitching algorithm according to the change of the real-time angle a, and then perform 360-degree surround stitching on the acquired front, rear, left and right view images. The image stitching algorithm is a prior art and will not be described here.
[0024] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0025] Please also see Figure 2The embodiment of the present invention further provides a method for real-time acquisition of a tractor trailer angle, comprising the following steps:
[0026] S1, obtaining the angular velocity of the trailer 2 measured by the second IMU 21 and recorded as av2, and obtaining the angular velocity of the tractor head 1 measured by the first IMU 11 and recorded as av1;
[0027] S2, obtaining the angular velocity difference diff_av=(av1-av2) between the tractor head 1 and the trailer 2;
[0028] S3, obtaining the relative rotation angle delt_a between the tractor head 1 and the trailer 2 at the current moment through the formula: delt_a=diff_av / f;
[0029] S4, accumulating the relative rotation angle delt_a of each frame, and then obtaining the angle a=Σdelt_a between the tractor head 1 and the trailer 2.
[0030] In step S1, when the vehicle starts, the acquisition frequency f of the first IMU 11 and the second IMU 21 needs to be further synchronized. The acquisition frequency f can be set according to actual needs. Preferably, the acquisition frequency f is 5 to 20 seconds. -1 In one embodiment, the acquisition frequency f is 10s -1 , that is, 10 times per second.
[0031] After step S4, the method further comprises:
[0032] S5, when the absolute value of av1 measured by the first IMU11 is close to 0, and the absolute value of av2 measured by the second IMU21 is also close to 0, it is determined that the tractor head 1 and the trailer 2 have been stretched to be straight, and the angle between the tractor head 1 and the trailer 2 is reset to a=0, and the process returns to step S1 to recalculate.
[0033] Please also see Figure 3 The embodiment of the present invention further provides a 360-degree surround view stitching method for a tractor trailer, comprising the following steps:
[0034] S1, obtaining the angular velocity of the trailer 2 measured by the second IMU 21 and recorded as av2, obtaining the angular velocity of the tractor head 1 measured by the first IMU 11 and recorded as av1, and obtaining the front, rear, left and right views of the vehicle in real time;
[0035] S2, obtaining the angular velocity difference diff_av=(av1-av2) between the tractor head 1 and the trailer 2;
[0036] S3, obtaining the relative rotation angle delt_a between the tractor head 1 and the trailer 2 at the current moment through the formula: delt_a=diff_av / f;
[0037] S4, accumulating the relative rotation angle delt_a of each frame, and then obtaining the angle a=Σdelt_a between the tractor head 1 and the trailer 2;
[0038] S6, dynamically adjusting the entire image stitching algorithm according to the change of the angle a, and then performing 360-degree surround stitching on the acquired front, rear, left, and right view images.
[0039] The present invention further provides a computer-readable storage medium having a data processing program stored thereon, and the data processing program can be executed by one or more processors to implement the steps of the above-mentioned method for real-time acquisition of trailer angles or the method for 360-degree surround view stitching of a tractor trailer.
[0040] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0041] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0042] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0043] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A 360-degree surround stitching method for a tractor trailer, the tractor trailer comprising a tractor head and a trailer; a first IMU is further arranged at a middle position on the top of the tractor head, and a second IMU is further arranged on the trailer to be communicatively connected to the first IMU; the first IMU and the second IMU are arranged on a straight line parallel to the length direction of the vehicle, the second IMU is arranged on the trailer, and the second IMU is arranged at a middle position of the rear of the roof of the trailer, the X-axis, Y-axis and Z-axis of the first IMU are respectively oriented in the same direction as the X-axis, Y-axis and Z-axis of the second IMU; and a camera for obtaining front, rear, left and right views of the vehicle, the method comprising the following steps: S1, obtaining the angular velocity of the trailer measured by the second IMU and recorded as av2, obtaining the angular velocity of the tractor head measured by the first IMU and recorded as av1, and obtaining the front, rear, left and right views of the vehicle in real time; S2, obtaining the angular velocity difference between the tractor head and the trailer, diff_av=(av1-av2); S3, obtaining the relative rotation angle delt_a between the tractor head and the trailer at the current moment through the formula: delt_a=diff_av / f; S4, accumulating the relative rotation angle del t_a of each frame, and then obtaining the angle a=Σdel t_a between the tractor head and the trailer; S5, dynamically adjusting the entire image stitching algorithm according to the change of the angle a, and then performing 360-degree surround stitching on the acquired front, rear, left, and right view images.
2. The 360-degree surround view splicing method for a tractor trailer according to claim 1, characterized in that: In step S1, when the vehicle is started, the acquisition frequencies f of the first IMU and the second IMU need to be further synchronized.
3. The 360-degree surround view splicing method for a tractor trailer according to claim 2, characterized in that: The acquisition frequency f is 5 to 20 s -1 .
4. The 360-degree surround view stitching method for a tractor trailer according to claim 1, characterized in that: After step S4, the method further comprises: S5, when the absolute value of av1 measured by the first IMU is close to 0, and the absolute value of av2 measured by the second IMU is also close to 0, it is determined that the tractor head and the trailer have been stretched to be straight. At this time, the angle between the tractor head and the trailer is reset to zero, and the process returns to step S1 to calculate the angular velocity.
5. The 360-degree surround stitching method for a tractor trailer according to claim 1, characterized in that: The front, rear, left and right views of the vehicle are obtained by a first camera arranged on both sides of the front of the tractor head and a second camera arranged on both sides of the rear of the trailer.
6. The 360-degree surround view stitching method for a tractor trailer according to claim 5, characterized in that: The first IMU and the second IMU are arranged on a central axis along the length direction of the vehicle.
7. The 360-degree surround view stitching method for a tractor trailer according to claim 5, characterized in that: The first camera and the second camera are both fisheye cameras.
Citation Information
Patent Citations
Method and device for determining included angle between truck head and trailer
CN113158864A
360-degree look-around system of trailer
CN210234807U
Method and controller for determining a trailer orientation
WO2019242988A1
Method for measuring angle between two body parts of foldable device, and device therefor
WO2021101171A1
