A method, electronic device, and storage medium for solving 3D panoramic distortion

Abstract

This invention discloses a method, electronic device, and storage medium for solving 3D panoramic distortion. It addresses 3D panoramic distortion by modifying the bowl model or adjusting the OpenGL viewpoint parameters while keeping them constant. The method provides two approaches: modifying the bowl model or adjusting the OpenGL viewpoint parameters. When modifying the bowl model, the model is changed in four areas: front, back, left, and right. Only the bowl model in areas with obstacles is modified, while other areas remain unchanged. Furthermore, the bowl model changes gradually to avoid abrupt changes and maintain display quality. When adjusting the OpenGL viewpoint parameters, the original 3D panoramic viewpoint parameters are gradually shifted to point towards the obstacle closest to the vehicle. This method is simple to operate, effective, and suitable for industrial application.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method, electronic device and storage medium for solving 3D panoramic distortion. Background Technology

[0002] 360-degree panoramic view is one of the most widely used technologies in automotive driver assistance systems. With technological advancements, 3D panoramic views have gradually replaced 2D panoramic views, becoming the mainstream. Compared to 2D panoramic views, which only display the ground scene around the car, 3D panoramic views construct a 3D model (called a bowl model because of its bowl-like shape) and draw the panoramic image onto it, displaying a much wider range of scenes. Using OpenGL, it can also achieve 360-degree horizontal rotation, allowing users to view the panoramic perspective from any angle. 3D panoramic views provide drivers with more information, more effectively preventing collisions and other hazards. However, current 3D panoramic views also have some drawbacks, such as the appearance of object distortion.

[0003] A 3D panoramic bowl model can be divided into a bowl base (usually built on a plane with a height of 0) and bowl walls (with a height of not 0). The car model is placed in the center of the bowl base. When an obstacle around the car is partially drawn on the bowl base and partially on the bowl walls, the obstacle will appear distorted in the 3D panoramic display due to the change in height between the bowl base and the bowl walls. Therefore, designing a method to solve the distortion problem in 3D panoramic displays has significant practical implications.

[0004] Chinese invention patent, publication number CN113689552A, entitled "A Method, Device, Electronic Device, and Storage Medium for Adjusting a Vehicle Surround View Model," discloses: first, acquiring a vehicle surround view model; then, acquiring the lateral and longitudinal distances between the vehicle and obstacles; and finally, adjusting the vehicle surround view model based on the lateral and longitudinal distances. While this patent can solve the distortion problem in 3D panoramas, it has the following drawbacks:

[0005] 1) When an obstacle is detected, the size of the bowl model suddenly changes, and the 3D panoramic image will jump once. When obstacle detection is unstable or there are many obstacles, the 3D panoramic image may jump frequently, resulting in a poor visual effect.

[0006] 2) When an obstacle is detected, the bowl model changes as a whole without distinguishing between regions. The area where the obstacle is located will no longer be distorted, but it may affect other areas and make the display effect of other areas worse. Summary of the Invention

[0007] To address the technical problems existing in the prior art, the present invention aims to provide a method, electronic device, and storage medium for solving 3D panoramic distortion.

[0008] To achieve the above objectives and technical effects, the technical solution adopted by this invention is as follows:

[0009] One method to solve 3D panoramic distortion is to change the bowl model or adjust the OpenGL viewpoint parameters while keeping the OpenGL viewpoint parameters unchanged.

[0010] Furthermore, the steps to resolve 3D panoramic distortion by modifying the bowl model while keeping the OpenGL viewpoint parameters constant include:

[0011] 1) Draw the panorama using the default bowl model;

[0012] 2) Based on the radar detection results, determine whether there are obstacles in the four zones of the panorama: front, rear, left, and right. If there are obstacles, compare the distance from the obstacle to the vehicle body with the radius of the current bowl model. If it is less than the radius of the bowl model but greater than the threshold, gradually change the radius of the ellipse in that direction to the distance from the obstacle to the vehicle body, and update the bowl model and panorama display.

[0013] 3) If there are no obstacles in a certain partition for a continuous period of time, the radius of the ellipse in that direction will be restored to the default value;

[0014] 4) Repeat steps 2)-3).

[0015] Furthermore, in step 1), the bottom of the bowl model is elliptical with a minor radius of a, a>0, and a major radius of b, b>0. The seam of the panorama divides the panorama into four sections: front, back, left, and right.

[0016] Furthermore, in step 2), the rule for comparing the distance from the obstacle to the vehicle body with the radius of the current bowl model is as follows:

[0017] Comparison of the absolute Y-coordinate of obstacles in the front and rear sections with the major radius of the ellipse;

[0018] Comparison of the absolute X-coordinates of obstacles in the left and right sections with the minor radius of the ellipse.

[0019] Furthermore, in step 2), if there are obstacles in the area in front of or behind the vehicle, let the coordinates of the obstacles be (obj). x ,obj y If the absolute value of the Y-coordinate of the obstacle is |obj, then the absolute value of the Y-coordinate of the obstacle will be |obj. y |Compared with the major radius b of the ellipse in the current bowl model, if |obj y If | is less than b and greater than the threshold, then the major radius b in that direction is gradually replaced with |obj. y | Update the bowl model and panoramic display;

[0020] To avoid abrupt changes in the panoramic image, the long radius 'b' should be gradually changed to |obj.y |;

[0021] Let N be the number of gradations, and let N be adjusted according to the hardware processing capabilities and desired effect. Then the step size for each gradation is:

[0022] step = (|obj) y |-b) / N

[0023] The more times the gradient is applied, the smoother the panorama changes. The bowl model needs to be transformed before the next frame arrives, so the value of N must be a trade-off between the effect and the time consumption.

[0024] The bowl model cannot be infinitely reduced in size if |obj y If the value is less than the preset threshold, the original bowl model will remain unchanged.

[0025] Furthermore, in step 2), if there is an obstacle in the left or right section of the vehicle, let the coordinates of the obstacle be (obj). x ,obj y If the absolute value of the obstacle's X coordinate is |obj, then the absolute value of the obstacle's X coordinate is |obj. x |Compared with the minor radius 'a' of the ellipse in the current bowl model, if |obj x If | is less than a and greater than the threshold, then the short radius a in that direction is gradually replaced with |obj|. x | Update the bowl model and panoramic display;

[0026] To avoid abrupt changes in the panoramic image, the short radius 'a' should be gradually changed to |obj. x |;

[0027] Let N be the number of gradations, and let N be adjusted according to the hardware processing capabilities and desired effect. Then the step size for each change is:

[0028] step = (|obj) x |-a) / N

[0029] The more times the gradient is applied, the smoother the panorama changes. The bowl model needs to be transformed before the next frame arrives, so the value of N must be a trade-off between the effect and the time consumption.

[0030] The bowl model cannot be infinitely reduced in size if |obj x If the value is less than the preset threshold, the original bowl model will remain unchanged.

[0031] Furthermore, the steps to resolve 3D panoramic distortion by adjusting OpenGL viewpoint parameters include:

[0032] 1) Initially, the default panoramic view or the view manually selected by the customer is displayed;

[0033] 2) If it is in drive, check for obstacles in front of the vehicle; if it is in reverse, check for obstacles behind the vehicle; for other gears, check for obstacles around the vehicle.

[0034] 3) If an obstacle is detected, the original 3D panoramic view parameters will be gradually changed to point towards the obstacle closest to the vehicle;

[0035] 4) During steps 2)-3), if the customer manually selects a viewpoint, the customer's selected viewpoint will be displayed first.

[0036] Furthermore, in step 3), let the original 3D panoramic viewpoint parameter be (eye). x0 eye y0 eye z0 The world coordinates of the obstacle are (obj) x ,obj y The new viewpoint parameter (eye) is then... x1 eye y1 eye z1 )for:

[0037]

[0038] In the formula, R is the radius of eye rotation;

[0039] To prevent abrupt changes in the panoramic image, (eye) x0 eye y0 eye z0 It should also gradually change to (eye) x1 eye y1 eye z1 );

[0040] Let C be the number of transitions, then the step size for each transition is:

[0041]

[0042] The present invention also discloses an electronic device, comprising:

[0043] At least one processor; and

[0044] A memory that is communicatively connected to the at least one processor;

[0045] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.

[0046] The present invention also discloses a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method described above.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] This invention discloses a method, electronic device, and storage medium for solving 3D panoramic distortion. It addresses the problem by altering the bowl model or adjusting the OpenGL viewpoint parameters while maintaining the OpenGL viewpoint parameters constant. When altering the bowl model, the model is changed in four regions: front, back, left, and right. Only the bowl model in the region containing obstacles is modified, while other regions remain unchanged. Furthermore, the bowl model changes gradually to avoid abrupt changes and maintain display quality. When adjusting the OpenGL viewpoint parameters, the original 3D panoramic viewpoint parameters are gradually shifted to point towards the obstacle closest to the vehicle. This method is simple to operate, effective, and suitable for industrial application. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the bowl model of the present invention;

[0050] Figure 2 This is a schematic diagram of the elliptical bottom of the bowl in Embodiment 1 of the present invention;

[0051] Figure 3 This is a schematic diagram showing the distortion of an object in a panorama in Embodiment 1 of the present invention;

[0052] Figure 4 This is a schematic diagram illustrating the method of eliminating distortion by changing the shape of the bowl model in Embodiment 1 of the present invention;

[0053] Figure 5 This is a flowchart of Embodiment 1 of the present invention;

[0054] Figure 6 This is a schematic diagram of the distortion at the vehicle tires on the right front in the panoramic view of Embodiment 1 of the present invention;

[0055] Figure 7 This is a schematic diagram showing the disappearance of distortion after transforming the bowl model according to radar information in Embodiment 1 of the present invention;

[0056] Figure 8 This is a schematic diagram of the OpenGL perspective in Embodiment 2 of the present invention;

[0057] Figure 9-10 These are schematic diagrams showing the distortion of the tires of the obstacle in Embodiment 2 of the present invention, where the viewing direction is not towards the right front of the car.

[0058] Figure 11 This is a schematic diagram showing the disappearance of tire distortion of the obstacle when the viewing direction is towards the right front of the car in Embodiment 2 of the present invention. Figure 12 This is a schematic diagram of OpenGL perspective adjustment in Embodiment 2 of the present invention. Detailed Implementation

[0059] The present invention will now be described in detail so that its advantages and features can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0060] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0061] Example 1

[0062] like Figure 1-7 As shown, a method for solving 3D panoramic distortion includes the following steps:

[0063] 1) Draw the panorama using the default bowl model;

[0064] Bowl models commonly used in 3D panoramas, such as Figure 1 As shown, the bottom of the bowl model is generally elliptical or rounded rectangle, and the bottom of the bowl model in this embodiment 1 is elliptical.

[0065] Let the minor radius of the ellipse be *a*, where *a* > 0, and the major radius be *b*, where *b* > 0. The seam of the panorama divides the panorama area into four sections: front, back, left, and right. Figure 2 As shown;

[0066] When there is an object at the edge of the ellipse, such as Figure 3 As shown, taking the area in front of the car as an example, the object is partially drawn on the bottom of the bowl and partially on the bowl wall, resulting in distortion in the 3D panorama. The position of the object is detected using radar, and its coordinates are set as (objx, objy). The bowl model is then modified to adjust the absolute value of the obstacle's Y-coordinate |obj... y |Compared with the major radius b of the ellipse in the current bowl model, if |obj y If | is less than b and greater than the threshold, gradually change the intersection point of the ellipse with the positive Y-axis from b to |obj. y |, while keeping the intersection points with the other three axes unchanged, update the bowl model and panoramic display, such as Figure 4 As shown, the object is then drawn entirely on the bowl wall, eliminating distortion. When only obstacles in front of the vehicle are distorted in the panoramic view, this method only changes the shape of the bowl model in front of the vehicle, eliminating distortion, without affecting the display of the rear of the vehicle.

[0067] To avoid abrupt changes in the panoramic image, 'b' should be gradually replaced with |obj. y |;

[0068] Let N be the number of gradations, and let N be adjusted according to the hardware processing capabilities and desired effect. Then the step size for each gradation is:

[0069] step = (|obj) y |-b) / N

[0070] The more gradients there are, the smoother the panorama changes, but the greater the computational load and the longer it takes. Since the bowl model needs to change before the next frame arrives, the value of N must be a trade-off between the effect and the time consumption.

[0071] The bowl model cannot be infinitely reduced in size if |obj y If the value is less than the preset threshold th (which is adjusted based on experience and actual results), the original bowl model remains unchanged.

[0072] The same applies to other areas; all four areas will be adjusted simultaneously, specifically as follows:

[0073] If there is an obstacle in the rear section of the vehicle, let the coordinates of the obstacle be (obj). x ,obj y If the absolute value of the Y-coordinate of the obstacle is |obj, then the absolute value of the Y-coordinate of the obstacle will be |obj. y |Compared with the major radius b of the ellipse in the current bowl model, if |obj y If | is less than b and greater than the threshold, then the major radius b in that direction is gradually replaced with |obj. y | Update the bowl model and panoramic display;

[0074] To avoid abrupt changes in the panoramic image, the long radius 'b' should be gradually changed to |obj. y |;

[0075] Let N be the number of gradations, and let N be adjusted according to the hardware processing capabilities and desired effect. Then the step size for each gradation is:

[0076] step = (|obj) y |-b) / N

[0077] The more times the gradient is applied, the smoother the panorama changes. The bowl model needs to be transformed before the next frame arrives, so the value of N must be a trade-off between the effect and the time consumption.

[0078] The bowl model cannot be infinitely reduced in size if |obj y If the value is less than the preset threshold, the original bowl model remains unchanged.

[0079] If there is an obstacle in the left or right lane of the vehicle, let the coordinates of the obstacle be (obj). x ,obj y If the absolute value of the obstacle's X coordinate is |obj, then the absolute value of the obstacle's X coordinate is |obj. x|Compared with the minor radius 'a' of the ellipse in the current bowl model, if |obj x If | is less than a and greater than the threshold, then the short radius a in that direction is gradually replaced with |obj|. x | Update the bowl model and panoramic display;

[0080] To avoid abrupt changes in the panoramic image, the short radius 'a' should be gradually changed to |obj. x |;

[0081] Let N be the number of gradations, and let N be adjusted according to the hardware processing capabilities and desired effect. Then the step size for each change is:

[0082] step = (|obj) x |-a) / N

[0083] The more times the gradient is applied, the smoother the panorama changes. The bowl model needs to be transformed before the next frame arrives, so the value of N must be a trade-off between the effect and the time consumption.

[0084] The bowl model cannot be infinitely reduced in size if |obj x If the value is less than the preset threshold, the original bowl model will remain unchanged.

[0085] In summary, this embodiment solves 3D panoramic distortion by changing the bowl model, while keeping the OpenGL view unchanged. First, the panorama is drawn using the default bowl model. Based on radar detection results, it is determined whether there are obstacles in the four zones: front, rear, left, and right. If there are obstacles, the distance from the obstacle to the vehicle body is compared with the radius of the ellipse in the current bowl model (the absolute value of the Y-coordinate of the obstacle in the front and rear zones is compared with the major radius, and the absolute value of the X-coordinate of the obstacle in the left and right zones is compared with the minor radius). If the distance is less than the bowl model radius but greater than a threshold (separate thresholds are set for the front and rear zones and the left and right zones), the ellipse radius in that direction is gradually replaced with the distance from the obstacle to the vehicle body, and the bowl model and panoramic display are updated. If there are no obstacles in a certain zone for 2 consecutive seconds, the ellipse radius in that direction is restored to the default value.

[0086] This embodiment also discloses an electronic device, including:

[0087] At least one processor; and

[0088] A memory that is communicatively connected to the at least one processor;

[0089] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.

[0090] This embodiment also discloses a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method described above.

[0091] Example 2

[0092] like Figure 8-12 As shown, a method for solving 3D panoramic distortion includes the following steps:

[0093] Besides changing the bowl model, the 3D panoramic distortion problem can also be solved by adjusting the OpenGL view parameters.

[0094] Displaying a 3D panorama requires setting OpenGL view parameters, mainly including where you are looking (eye coordinates). x eye y eye z And where to look (center coordinates) x ,center y ,center z These coordinates are all in the OpenGL coordinate system. In the OpenGL coordinate system, the positive X-axis points to the right, the positive Y-axis points to the sky, and the positive Z-axis points backward.

[0095] Generally, 3D panoramic view parameters are calculated by looking from the perimeter towards the model's origin. The "eye" parameter rotates around the center of the car, while the "center" parameter keeps the center of the car model constant, thus achieving a 360-degree panoramic view. For example... Figure 7 As shown in the figure, the ellipse is the bottom ellipse of the bowl, the black dot represents the eye position, the solid arrow points to the center position, indicating the direction of the line of sight, the radius of eye rotation is R, and the dashed circle is the trajectory of eye rotation.

[0096] Observations revealed that when obstacles appear distorted in a 3D panorama, the distortion disappears if the viewing angle is directed towards the obstacle. For example... Figure 9-11 As shown, with the bowl model unchanged, Figure 9-10 Because the viewing angle is not directed towards the vehicle on its right front, the tires of the obstacle appear distorted. Figure 11 With the viewpoint directed towards the right front of the car, the tire distortion of the obstacle disappears.

[0097] Based on this phenomenon, the viewing angle parameters can be changed by using the location of obstacles detected by radar, switching from the original viewing angle to a viewing angle facing the obstacle, thereby solving the distortion problem of 3D panoramas. Specific steps include:

[0098] 1) Initially, the default panoramic view or the view manually selected by the customer is displayed;

[0099] 2) If it is in drive, check for obstacles in front of the vehicle; if it is in reverse, check for obstacles behind the vehicle; for other gears, check for obstacles around the vehicle.

[0100] 3) If an obstacle is detected, the 3D panoramic view parameters are gradually adjusted to point towards the obstacle closest to the vehicle;

[0101] 4) During steps 2)-3), if the customer manually selects a viewpoint, the customer's selected viewpoint will be displayed first.

[0102] Let the original viewpoint parameter be (eye). x0 eye y0 eye z0 The world coordinates of the obstacle are (obj) x ,obj y ),like Figure 12 As shown, the new viewpoint parameters are:

[0103]

[0104] This formula includes the transformation from the world coordinate system to the OpenGL coordinate system.

[0105] Similarly, to prevent abrupt changes in the panoramic image, (eye) x0 eye y0 eye z0 It should also gradually change to (eye) x1 eye y1 eye z1 Let C be the number of gradations, then the step size for each gradation is:

[0106]

[0107] Any parts or structures not specifically described in this invention can be made using existing technologies or products, and will not be elaborated upon here.

[0108] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for solving 3D panoramic distortion, characterized in that, 3D panoramic distortion can be resolved by changing the bowl model or by adjusting the OpenGL view parameters while keeping the OpenGL view parameters constant. The steps to resolve 3D panoramic distortion by modifying the bowl model while keeping the OpenGL viewpoint parameters constant include: 1) Draw the panorama using the default bowl model; 2) Based on the radar detection results, determine whether there are obstacles in the four zones of the panorama: front, rear, left, and right. If there are obstacles, compare the distance from the obstacle to the vehicle body with the radius of the current bowl model. If it is less than the radius of the bowl model but greater than the threshold, gradually change the radius of the ellipse in the corresponding direction to the distance from the obstacle to the vehicle body, and update the bowl model and panorama display. 3) If there are no obstacles in a certain partition for a continuous period of time, the radius of the ellipse in the corresponding direction will be restored to the default value; 4) Repeat steps 2)-3). In step 1), the bottom of the bowl model is elliptical with a minor radius of a, a>0 and a major radius of b, b>0. The seam of the panorama divides the panorama into four sections: front, back, left, and right. In step 2), if there are obstacles in the area in front of or behind the vehicle, let the coordinates of the obstacles be (obj). x ,obj y If the absolute value of the Y-coordinate of the obstacle is |obj, then the absolute value of the Y-coordinate of the obstacle will be |obj. y |Compared with the major radius b of the ellipse in the current bowl model, if |obj y If | is less than b and greater than the threshold, then the major radius b in the corresponding direction is gradually replaced with |obj. y | Update the bowl model and panoramic display; To avoid abrupt changes in the panoramic image, the long radius 'b' should be gradually changed to |obj. y |; Let N be the number of gradations, and let N be adjusted according to the hardware processing capabilities and desired effect. Then the step size for each gradation is: ； The more times the gradient is applied, the smoother the panorama changes. The bowl model needs to be transformed before the next frame arrives, so the value of N must be a trade-off between the effect and the time consumption. The bowl model cannot be infinitely reduced in size if |obj y If the value is less than the preset threshold, the original bowl model will remain unchanged.

2. The method for solving 3D panoramic distortion according to claim 1, characterized in that, In step 2), the rule for comparing the distance from the obstacle to the vehicle body with the radius of the current bowl model is as follows: Comparison of the absolute Y-coordinate of obstacles in the front and rear sections with the major radius of the ellipse; Comparison of the absolute X-coordinates of obstacles in the left and right sections with the minor radius of the ellipse.

3. The method for solving 3D panoramic distortion according to claim 1, characterized in that, In step 2), if there is an obstacle in the left or right section of the vehicle, let the coordinates of the obstacle be (obj). x ,obj y If the absolute value of the obstacle's X coordinate is |obj, then the absolute value of the obstacle's X coordinate is |obj x |Compared with the minor radius 'a' of the ellipse in the current bowl model, if |obj x If | is less than a and greater than the threshold, then the short radius 'a' in the corresponding direction is gradually replaced with |obj|. x | Update the bowl model and panoramic display; To avoid abrupt changes in the panoramic image, the short radius 'a' should be gradually changed to |obj. x |; Let N be the number of gradations, and let N be adjusted according to the hardware processing capabilities and desired effect. Then, let the step size of each gradation be... for: ； The more times the gradient is applied, the smoother the panorama changes. The bowl model needs to be transformed before the next frame arrives, so the value of N must be a trade-off between the effect and the time consumption. The bowl model cannot be infinitely reduced in size if |obj x If the value is less than the preset threshold, the original bowl model will remain unchanged.

4. The method for solving 3D panoramic distortion according to claim 1, characterized in that, The steps to correct 3D panoramic distortion by adjusting OpenGL viewpoint parameters include: 1) Initially, the default panoramic view or the view manually selected by the customer is displayed; 2) If it is in drive, check for obstacles in front of the vehicle; if it is in reverse, check for obstacles behind the vehicle; for other gears, check for obstacles around the vehicle. 3) If an obstacle is detected, the original 3D panoramic view parameters will be gradually changed to point towards the obstacle closest to the vehicle; 4) During steps 2)-3), if the customer manually selects a viewpoint, the viewpoint selected by the customer will be displayed first.

5. A method for solving 3D panoramic distortion according to claim 4, characterized in that, In step 3), let the original 3D panoramic viewpoint parameter be (eye). x0 eye y0 eye z0 The world coordinates of the obstacle are (obj) x ,obj y The new viewpoint parameter (eye) is then... x1 eye y1 eye z1 )for: ； In the formula, R is the radius of eye rotation; To prevent abrupt changes in the panoramic image, (eye) x0 eye y0 eye z0 It should also gradually change to (eye) x1 eye y1 eye z1 ); Let C be the number of transitions, then the step size for each transition is: 。 6. An electronic device, characterized in that, include: At least one processor; A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-5.

7. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

Images