Vehicle side view generation method and device and vehicle

Abstract

The invention provides a vehicle side view generation method and device and a vehicle, and relates to the technical field of automobiles. The method comprises the following steps: converting a plurality of coordinates under a fisheye camera coordinate system into a plurality of coordinates under a unit spherical coordinate system; the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed in the vehicle, and the distances between coordinate points corresponding to multiple coordinates in the unit spherical coordinate system and the original point of the unit spherical coordinate system are the same; converting the plurality of coordinates under the unit spherical coordinate system into a plurality of coordinates under the virtual camera coordinate system based on the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system; the virtual camera coordinate system is a camera coordinate system corresponding to a virtual camera meeting the target acquisition view angle; and based on the internal parameters of the fisheye camera and the plurality of coordinates in the virtual camera coordinate system, performing projection to obtain a vehicle side view meeting the target acquisition visual angle. The vehicle side view generated by the method conforms to the intuition of human visual sense, so that the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of automobile technology, and in particular to a vehicle side view generation method, device, electronic equipment and storage medium. Background Art

[0002] With the rapid development of science and technology, AVM (Around View Monitor) is increasingly used in vehicles. AVM is a system that uses multiple ultra-wide-angle fisheye cameras to capture images, and then performs distortion correction and splicing on the captured images to form a panoramic image around the vehicle. Therefore, in order to form a panoramic image around the vehicle, it is necessary to generate a side view of the vehicle.

[0003] At present, for the left and right wheel side views of the vehicle, based on the external parameters of the fisheye camera, the original fisheye camera image captured by the fisheye camera is projected onto a bird's-eye view, and then the left and right wheel side views are generated after distortion processing and perspective transformation. However, in the generated left and right wheel side views, the objects around the vehicle are curved, and the ground around the vehicle is not on a straight horizon but is rotating. As a result, the generated left and right wheel side views will look particularly dizzy, which does not conform to the intuition of human visual senses, thus affecting the user experience. Summary of the invention

[0004] The present invention provides a vehicle side view generation method, device and vehicle, which are used to solve the defect of poor vehicle side view generation effect in the prior art and realize the generation of vehicle side images that meet the intuitive visual sensory perception of human beings.

[0005] The present invention provides a method for generating a vehicle side view, comprising: Convert multiple coordinates in a fisheye camera coordinate system into multiple coordinates in a unit spherical coordinate system; the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed on a vehicle, and the distances between coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system are all the same; Based on the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system, multiple coordinates in the unit sphere coordinate system are converted into multiple coordinates in the virtual camera coordinate system; the virtual camera coordinate system is a camera coordinate system corresponding to the virtual camera that satisfies the target acquisition viewing angle; Based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, a side view of the vehicle that meets the target acquisition viewing angle is projected.

[0006] According to a vehicle side view generation method provided by the present invention, the multiple coordinates in the unit spherical coordinate system all satisfy the following constraints: ; in,( , , ) is the coordinate in the unit spherical coordinate system; R is the preset radius length.

[0007] According to a vehicle side view generation method provided by the present invention, the fisheye camera coordinate system and the unit sphere coordinate system have the same origin position; and the coordinate axis directions of the fisheye camera coordinate system and the unit sphere coordinate system are the same.

[0008] According to a vehicle side view generating method provided by the present invention, the origin position of the virtual camera coordinate system is the same as that of the unit sphere coordinate system.

[0009] According to a vehicle side view generation method provided by the present invention, the vehicle side view satisfying the target acquisition viewing angle is projected based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, including: Based on the internal parameters of the fisheye camera, converting the multiple coordinates in the virtual camera coordinate system into multiple pixel coordinates corresponding to the fisheye camera; Based on the multiple pixel coordinates, a vehicle side view that meets the target acquisition viewing angle is projected.

[0010] According to a vehicle side view generation method provided by the present invention, projecting a vehicle side view satisfying the target acquisition viewing angle based on the multiple pixel coordinates includes: Based on the plurality of pixel coordinates, projecting to obtain an initial image; The initial image is captured based on the target acquisition angle of view to obtain a vehicle side view that meets the target acquisition angle of view.

[0011] The present invention also provides a vehicle side view generating device, comprising: A first conversion module is used to convert multiple coordinates in a fisheye camera coordinate system into multiple coordinates in a unit spherical coordinate system; the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed on a vehicle, and the distances between the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system are all the same; A second conversion module is used to convert multiple coordinates in the unit spherical coordinate system into multiple coordinates in the virtual camera coordinate system based on the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system; the virtual camera coordinate system is a camera coordinate system corresponding to the virtual camera that meets the target acquisition viewing angle; The view projection module is used to project a side view of the vehicle that meets the target acquisition viewing angle based on the internal parameters of the fisheye camera and multiple coordinates in the virtual camera coordinate system.

[0012] The present invention also provides a car, comprising: A fisheye camera, the fisheye camera is used to collect a fisheye image of a vehicle side of the vehicle; A processor is used to execute a program to implement any of the above-mentioned vehicle side view generation methods.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described vehicle side view generation methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the vehicle side view generating method as described in any one of the above is implemented.

[0015] The vehicle side view generation method, device and vehicle provided by the present invention convert multiple coordinates in a fisheye camera coordinate system into multiple coordinates in a unit sphere coordinate system, and the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed on the vehicle, and because the distances between the coordinate points corresponding to the multiple coordinates in the unit sphere coordinate system and the origin of the unit sphere coordinate system are the same, the multiple coordinates in the unit sphere coordinate system can be converted into multiple coordinates in the virtual camera coordinate system based on the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system, and the virtual camera coordinate system is a camera coordinate system corresponding to a virtual camera that meets the target acquisition viewing angle, so that based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, a vehicle side view that meets the target acquisition viewing angle can be projected without distortion processing and perspective transformation processing of near large and far small, so that while ensuring the generation of the vehicle side view that meets the target acquisition viewing angle, the ground around the vehicle in the generated vehicle side view can be seen on a straight horizon, and the objects around the vehicle in the generated vehicle side view are normal, thereby meeting the intuition of human visual senses and ultimately improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is one of the flow charts of the vehicle side view generating method provided by the present invention.

[0018] Figure 2This is the second flow chart of the vehicle side view generating method provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of a vehicle side view generating device provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] At present, for the left and right wheel side views of the vehicle, based on the external parameters of the fisheye camera, the original fisheye camera image captured by the fisheye camera is projected onto a bird's-eye view, and then distortion processing and perspective transformation are performed to generate the left and right wheel side views.

[0023] Specifically, set the world coordinate system and fisheye camera coordinate system There is a rotation and translation relationship between the world coordinate system and the fisheye camera coordinate system; based on this, the coordinates in the fisheye camera coordinate system can be converted to coordinates in the world coordinate system through the extrinsic rotation matrix and extrinsic translation vector of the fisheye camera, and then the wheel side view can be projected based on the intrinsic matrix of the fisheye camera and the coordinates in the world coordinate system.

[0024] For ease of understanding, a specific example is used here for illustration. Based on the external parameters of the fisheye camera, the original fisheye camera image captured by the fisheye camera is projected onto a bird's-eye view, that is, the original fisheye camera image is projected onto an image from a bird's-eye view in the world coordinate system; although the ground markings on the ground plane conform to human visual sensory intuition when the original fisheye camera image is projected onto an image from a bird's-eye view in the world coordinate system, the ground positions of green shrubs, trees, poles, and buildings on the curbs above the ground plane are abnormally stretched to the outside of the screen. Based on this, after the original fisheye camera image captured by the fisheye camera is projected onto a bird's-eye view, it will be distorted, for example, some distortion is introduced into the intrinsic parameter matrix of the fisheye camera (such as multiplying the focal length by a coefficient), so as to project a distorted image. In addition, perspective transformation processing can be performed to make objects larger when near and smaller when far away, so that the wheel side view is as consistent as possible with the intuition of the human eye. For example, a perspective transformation processing is performed on an image from a bird's-eye view without distortion to make objects larger when near and smaller when far away, to obtain an image after perspective transformation, or a perspective transformation processing is performed on an image with distortion to make objects larger when near and smaller when far away, to obtain an image after perspective transformation. However, the objects around the vehicle in the generated wheel side view are curved, that is, the green shrubs, trees, poles, parked vehicles on the left side of the road that are above the ground level, and the surrounding buildings are all curved at an angle, and the land around the vehicle is not on a straight horizon but is rotating, so the generated left wheel side view will look particularly dizzy, which does not conform to the intuition of human visual senses, thereby affecting the user experience.

[0025] In view of the above problems, the present invention proposes the following embodiments. Figure 1-Figure 2 The vehicle side view generation method of the present invention is described.

[0026] Figure 1 is one of the flow charts of the vehicle side view generation method provided by the present invention, such as Figure 1 As shown, the vehicle side view generating method includes the following steps 110, 120 and 130.

[0027] Step 110: convert multiple coordinates in the fisheye camera coordinate system into multiple coordinates in the unit sphere coordinate system.

[0028] The fisheye camera coordinate system is a camera coordinate system corresponding to the fisheye camera deployed on the vehicle.

[0029] In one embodiment, the fisheye camera coordinate system is set The X-axis of the fisheye camera coordinate system is on the right side of the fisheye camera, the Y-axis is below the fisheye camera, and the Z-axis is in front of the fisheye camera, satisfying the right-hand rule, that is, the fisheye camera coordinate system corresponding to the fisheye camera is defined in the order of the bottom right front of the fisheye camera body.

[0030] Exemplarily, the fisheye camera deployed on the vehicle generally includes a left fisheye camera and a right fisheye camera, the left fisheye camera is deployed under the left rearview mirror of the vehicle, and the acquisition direction of the left fisheye camera is the lower left direction of the vehicle, and the right fisheye camera is deployed under the right rearview mirror of the vehicle, and the acquisition direction of the right fisheye camera is the lower right direction of the vehicle, so as to collect a complete side image of the vehicle (the image of the left side of the vehicle body and the image of the right side of the vehicle body). Of course, the fisheye camera deployed on the vehicle can also include a front fisheye camera and a rear fisheye camera, the front fisheye camera is deployed on the front side of the vehicle, and the rear fisheye camera is deployed on the rear side of the vehicle.

[0031] Here, when the fisheye camera original image is collected by the fisheye camera, multiple coordinates in the fisheye camera coordinate system can be obtained; that is, based on the multiple coordinates in the fisheye camera coordinate system and the internal parameters of the fisheye camera, the fisheye camera original image can be projected.

[0032] Among them, the distances between the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system are all the same. It should be noted that by converting the multiple coordinates in the fisheye camera coordinate system into the multiple coordinates in the unit spherical coordinate system, the distances between the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system can be the same. That is, the unit spherical coordinate system (normalized spherical coordinate system) can make the distances between the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system the same. For ease of understanding, for example, image projection is performed based on the multiple coordinates in the unit spherical coordinate system, that is, the original image of the fisheye camera is projected onto the unit sphere.

[0033] Furthermore, the origin position of the fisheye camera coordinate system is the same as that of the unit sphere coordinate system, so when converting multiple coordinates in the fisheye camera coordinate system into multiple coordinates in the unit sphere coordinate system, there is no need to perform translation based on the translation vector, thereby improving the efficiency of coordinate conversion and further improving the efficiency of generating the vehicle side view.

[0034] Furthermore, the coordinate axes of the fisheye camera coordinate system and the unit sphere coordinate system are in the same direction, so when converting multiple coordinates in the fisheye camera coordinate system into multiple coordinates in the unit sphere coordinate system, there is no need to rotate based on a rotation matrix, thereby improving the efficiency of coordinate conversion and further improving the efficiency of generating the vehicle side view.

[0035] Step 120: based on the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system, convert the multiple coordinates in the unit sphere coordinate system into the multiple coordinates in the virtual camera coordinate system.

[0036] The virtual camera coordinate system is a camera coordinate system corresponding to a virtual camera that satisfies the target acquisition viewing angle.

[0037] Here, the target acquisition angle of view may be the target direction expected to be seen from the vehicle side, that is, the target direction of the vehicle expected to be seen. For example, if a vehicle side view of the left front area of ​​the vehicle needs to be obtained, the target acquisition angle of view is the left front area of ​​the vehicle.

[0038] Here, the virtual camera is not a real deployed camera, but the final side view of the vehicle can be defaulted to the view captured by the virtual camera. Based on this, even if the left fisheye camera is deployed under the left rearview mirror of the vehicle, and the acquisition direction of the left fisheye camera is the lower left direction of the vehicle, and the right fisheye camera is deployed under the right rearview mirror of the vehicle, and the acquisition direction of the right fisheye camera is the lower right direction of the vehicle, although the acquisition angle of the fisheye camera is inconsistent with the target acquisition angle (such as the left front area of ​​the vehicle), there is no need to change the physical deployment posture of the left and right fisheye cameras, thereby reducing costs and ensuring that the lower area of ​​the vehicle (such as tires) can be photographed. Of course, other fisheye cameras do not need to change their physical deployment posture, that is, there is no need to adjust the physical deployment posture of the camera according to the target acquisition angle, which improves flexibility.

[0039] It should be noted that, considering that in the prior art, the bird's-eye view after projecting the original image of the fisheye camera to the world coordinate system is inconsistent with the actual view of the human eye, the generated side view of the vehicle will make people dizzy, that is, since the acquisition perspective of the fisheye camera is inconsistent with the target acquisition perspective of the virtual camera, the virtual camera coordinate system is inconsistent with the unit sphere coordinate system. Therefore, it is necessary to convert multiple coordinates in the unit sphere coordinate system into multiple coordinates in the virtual camera coordinate system based on the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system, that is, to obtain multiple coordinates corresponding to the virtual camera that meet the target acquisition perspective, to ensure that the vehicle side view that meets the target acquisition perspective is finally obtained, and the actual view effect captured by the camera that meets the target acquisition perspective is achieved.

[0040] Here, considering that the multiple coordinates in the unit spherical coordinate system are converted into multiple coordinates in the virtual camera coordinate system, the purpose is to convert the acquisition angle of the fisheye camera into the target acquisition angle. Based on this, the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system is determined based on the fisheye camera's posture and the target acquisition angle. Of course, the unit spherical coordinate system is pre-set, that is, fixed, so the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system can be dynamically updated based on the fisheye camera's posture and the target acquisition angle.

[0041] Furthermore, when the origin position of the fisheye camera coordinate system is the same as that of the unit sphere coordinate system, and the coordinate axis directions of the fisheye camera coordinate system and the unit sphere coordinate system are the same, the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system can be directly determined based on the posture of the fisheye camera and the target acquisition viewing angle, that is, the conversion relationship between the virtual camera coordinate system and the unit sphere coordinate system (that is, the fisheye camera coordinate system) can be determined directly based on the relationship between the posture of the fisheye camera and the target acquisition viewing angle.

[0042] In one embodiment, the origin position of the virtual camera coordinate system is the same as that of the unit spherical coordinate system. That is, the origin position of the virtual camera coordinate system is set to be the same as that of the unit spherical coordinate system, and then based on the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system, multiple coordinates in the unit spherical coordinate system can be converted into multiple coordinates in the virtual camera coordinate system without translation based on the translation vector, thereby improving the efficiency of coordinate conversion and thus improving the efficiency of generating the vehicle side view.

[0043] Exemplarily, the coordinate conversion formula for converting multiple coordinates in the unit sphere coordinate system into multiple coordinates in the virtual camera coordinate system is as follows: ; In the formula, is the coordinate in the virtual camera coordinate system, is the coordinate in the unit spherical coordinate system, Represents the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system.

[0044] It should be noted that, in fact, the position and posture of the unit spherical coordinate system is rotated, and because the distances between the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system are the same, the rotated unit spherical coordinate system can be directly used as the virtual camera coordinate system without resetting the virtual camera coordinate system. And it is precisely by rotating the position and posture of the unit spherical coordinate system that the multiple coordinates in the unit spherical coordinate system can be converted into multiple coordinates in the virtual camera coordinate system.

[0045] Exemplarily, any rotation operation can be added to the posture of the unit spherical coordinate system in the unit spherical coordinate system, so as to obtain the projection of the unit sphere on the virtual camera posture that satisfies the target acquisition viewing angle, that is, after projecting the coordinate point in the fisheye camera coordinate system onto the unit sphere, any rotation operation is added to the posture of the unit spherical coordinate system to reproject the projection of the unit sphere on the virtual camera posture that satisfies the target acquisition viewing angle. Since the distances between the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system are the same, when back-projecting back to the virtual camera coordinate system, the projection of the unit sphere on the virtual camera posture that satisfies the target acquisition viewing angle obtained by reprojection can be directly used as the projection in the virtual camera coordinate system.

[0046] In another embodiment, the origin positions of the virtual camera coordinate system and the unit spherical coordinate system are different. That is, the origin position of the virtual camera coordinate system is set to be different from the origin position of the unit spherical coordinate system, and then based on the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system and the translation vector of the virtual camera coordinate system relative to the unit spherical coordinate system, multiple coordinates in the unit spherical coordinate system are converted into multiple coordinates in the virtual camera coordinate system.

[0047] Step 130 , based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, project a side view of the vehicle that meets the target acquisition viewing angle.

[0048] Specifically, since the virtual camera coordinate system is the camera coordinate system corresponding to the virtual camera that satisfies the target acquisition angle, based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, the side view of the vehicle that satisfies the target acquisition angle can be projected. More specifically, based on the internal parameter matrix of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, the side view of the vehicle that satisfies the target acquisition angle is projected.

[0049] It should be understood that the embodiments of the present invention do not require distortion processing and perspective transformation processing where objects appear larger near and smaller far away, thereby achieving the goal of generating a side view of the vehicle in which the ground around the vehicle appears to be on a straight horizon, and that the target acquisition perspective can look far away, that is, in the generated side view of the vehicle, green shrubs, trees, poles, vehicles parked on the roadside, surrounding buildings, etc., which are above the ground level in the distance can be seen, and are straight above the horizon in accordance with the user's eye intuition, thereby improving the user experience.

[0050] The vehicle side view generation method provided by the embodiment of the present invention converts multiple coordinates in a fisheye camera coordinate system into multiple coordinates in a unit sphere coordinate system, and the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed on a vehicle, and because the distances between the coordinate points corresponding to the multiple coordinates in the unit sphere coordinate system and the origin of the unit sphere coordinate system are the same, the multiple coordinates in the unit sphere coordinate system can be converted into multiple coordinates in the virtual camera coordinate system based on the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system, and the virtual camera coordinate system is a camera coordinate system corresponding to a virtual camera that meets the target acquisition viewing angle, so that based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, a vehicle side view that meets the target acquisition viewing angle can be projected without distortion processing and perspective transformation processing of near large and far small, so that while ensuring the generation of a vehicle side view that meets the target acquisition viewing angle, the ground around the vehicle in the generated vehicle side view can be seen on a straight horizon, and the objects around the vehicle in the generated vehicle side view are normal, thereby meeting the intuition of human visual senses and ultimately improving the user experience.

[0051] Based on any of the above embodiments, in this method, the multiple coordinates in the unit spherical coordinate system all satisfy the following constraints: ; in,( , , ) is the coordinate in the unit spherical coordinate system; R is the preset radius length.

[0052] Here, the preset radius length R can be set according to actual needs, for example, 1.

[0053] It should be noted that, when converting multiple coordinates in the fisheye camera coordinate system into multiple coordinates in the unit sphere coordinate system, it is necessary to ensure that the coordinates in the converted unit sphere coordinate system satisfy the above constraints.

[0054] The vehicle side view generation method provided by an embodiment of the present invention ensures that multiple coordinates in the unit spherical coordinate system satisfy the above constraints, thereby ensuring that the multiple coordinates in the unit spherical coordinate system can be converted into multiple coordinates in the virtual camera coordinate system based on the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system, thereby ultimately improving the user experience.

[0055] Based on any of the above embodiments, in the method, the fisheye camera coordinate system and the unit sphere coordinate system have the same origin position; and the coordinate axis directions of the fisheye camera coordinate system and the unit sphere coordinate system are the same.

[0056] For example, the fisheye camera coordinate system is set The X-axis of the fisheye camera coordinate system is on the right side of the fisheye camera, the Y-axis is below the fisheye camera, and the Z-axis is in front of the camera, satisfying the right-hand rule, that is, the fisheye camera coordinate system corresponding to the fisheye camera is defined in the order of the bottom right front of the fisheye camera body; based on this, the unit spherical coordinate system is set , the X-axis of the unit spherical coordinate system is to the right of the fisheye camera, the Y-axis is to the bottom of the fisheye camera, and the Z-axis is to the front of the fisheye camera, so that the coordinate axis directions of the fisheye camera coordinate system and the unit spherical coordinate system are the same.

[0057] It should be understood that although the fisheye camera coordinate system and the unit spherical coordinate system have the same origin position and the same coordinate axis direction, the unit spherical coordinate system has a constraint, which can be "the distances between the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system are the same", or the above constraint " ”.

[0058] Exemplarily, when the fisheye camera coordinate system and the unit sphere coordinate system have the same origin position and the coordinate axis directions of the fisheye camera coordinate system and the unit sphere coordinate system are the same, a coordinate conversion formula for converting multiple coordinates in the fisheye camera coordinate system into multiple coordinates in the unit sphere coordinate system is as follows: ; In the formula, represents the coordinates in the unit spherical coordinate system, Represents the coordinates in the fisheye camera coordinate system.

[0059] It can be understood that the above method can be understood as blurring the image pixels on the original image of the fisheye camera onto a unit sphere, and the photo obtained by the fisheye camera observing the unit sphere in the current posture is consistent with the photo obtained by observing the original world.

[0060] In the vehicle side view generation method provided by the embodiment of the present invention, the origin position of the fisheye camera coordinate system is the same as that of the unit sphere coordinate system, so that when converting multiple coordinates in the fisheye camera coordinate system into multiple coordinates in the unit sphere coordinate system, there is no need to perform translation based on a translation vector, thereby improving the coordinate conversion efficiency, and further improving the vehicle side view generation efficiency; and the coordinate axis direction of the fisheye camera coordinate system and the unit sphere coordinate system is the same, so when converting multiple coordinates in the fisheye camera coordinate system into multiple coordinates in the unit sphere coordinate system, there is no need to perform rotation based on a rotation matrix, thereby improving the coordinate conversion efficiency, and further improving the vehicle side view generation efficiency.

[0061] Based on any of the above embodiments, Figure 2 FIG. 2 is a flow chart of the vehicle side view generation method provided by the present invention. Figure 2As shown, the above step 130 includes: step 131 and step 132.

[0062] Step 131 : based on the internal parameters of the fisheye camera, convert the multiple coordinates in the virtual camera coordinate system into multiple pixel coordinates corresponding to the fisheye camera.

[0063] Specifically, based on the intrinsic parameter matrix of the fisheye camera, multiple coordinates in the virtual camera coordinate system are converted into multiple pixel coordinates corresponding to the fisheye camera.

[0064] Exemplarily, the conversion formula for converting multiple coordinates in the virtual camera coordinate system into multiple pixel coordinates corresponding to the fisheye camera is as follows: ; In the formula, represents the pixel coordinates corresponding to the fisheye camera, K represents the intrinsic parameter matrix of the fisheye camera, Represents the coordinates in the virtual camera coordinate system.

[0065] Step 132: Based on the plurality of pixel coordinates, project a side view of the vehicle that satisfies the target acquisition viewing angle.

[0066] Specifically, since the virtual camera coordinate system is the camera coordinate system corresponding to the virtual camera that meets the target acquisition viewing angle, based on the internal parameters of the fisheye camera, after converting multiple coordinates in the virtual camera coordinate system into multiple pixel coordinates corresponding to the fisheye camera, the vehicle side view that meets the target acquisition viewing angle can be projected based on the multiple pixel coordinates.

[0067] The vehicle side view generation method provided by the embodiment of the present invention does not need to perform distortion processing and perspective transformation processing of near-big-far-small through the above-mentioned manner, so that the ground around the vehicle in the generated vehicle side view can be seen on a straight horizon, and the target acquisition perspective can look far away, that is, the green shrubs, trees, poles, roadside parked vehicles and surrounding buildings on the roadside above the ground level in the distance can be seen in the generated vehicle side view, and are straight above the horizon in line with the user's eye intuition, thereby improving the user experience.

[0068] Based on any of the above embodiments, considering that the area covered by the fisheye camera is relatively wide at the beginning, in this method, step 132 includes: Based on the plurality of pixel coordinates, projecting to obtain an initial image; The initial image is captured based on the target acquisition angle of view to obtain a vehicle side view that meets the target acquisition angle of view.

[0069] Specifically, if a complete initial image is obtained by projection, the initial image is intercepted based on the target acquisition angle to obtain a side view of the vehicle that meets the target acquisition angle. If multiple initial images are obtained by projection, the multiple initial images are first spliced ​​to obtain a complete image, and then the complete image is intercepted based on the target acquisition angle to obtain a side view of the vehicle that meets the target acquisition angle.

[0070] The vehicle side view generation method provided by an embodiment of the present invention projects an initial image based on multiple pixel coordinates in the above manner, and captures the initial image based on a target acquisition perspective to ensure that a vehicle side view that meets the target acquisition perspective is obtained, that is, the vehicle side view is limited to an image of an area corresponding to the target acquisition perspective, thereby ensuring subsequent view display effects and improving user experience.

[0071] The vehicle side view generating device provided by the present invention is described below. The vehicle side view generating device described below and the vehicle side view generating method described above can refer to each other.

[0072] Figure 3 is a schematic diagram of the structure of the vehicle side view generating device provided by the present invention, such as Figure 3 As shown, the vehicle side view generating device includes a first conversion module 310 , a second conversion module 320 and a view projection module 330 .

[0073] The first conversion module 310 is used to convert multiple coordinates in a fisheye camera coordinate system into multiple coordinates in a unit spherical coordinate system; the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed on a vehicle, and the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system are at the same distance from the origin of the unit spherical coordinate system.

[0074] The second conversion module 320 is used to convert multiple coordinates in the unit spherical coordinate system into multiple coordinates in the virtual camera coordinate system based on the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system; the virtual camera coordinate system is a camera coordinate system corresponding to the virtual camera that meets the target acquisition viewing angle.

[0075] The view projection module 330 is used to project a side view of the vehicle that meets the target acquisition viewing angle based on the internal parameters of the fisheye camera and multiple coordinates in the virtual camera coordinate system.

[0076] The vehicle side view generating device provided by the embodiment of the present invention converts multiple coordinates in the fisheye camera coordinate system into multiple coordinates in the unit sphere coordinate system, and the fisheye camera coordinate system is a camera coordinate system corresponding to the fisheye camera deployed on the vehicle, and because the distances between the coordinate points corresponding to the multiple coordinates in the unit sphere coordinate system and the origin of the unit sphere coordinate system are the same, the multiple coordinates in the unit sphere coordinate system can be converted into multiple coordinates in the virtual camera coordinate system based on the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system, and the virtual camera coordinate system is a camera coordinate system corresponding to the virtual camera that meets the target acquisition viewing angle, so that based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, a vehicle side view that meets the target acquisition viewing angle can be projected without distortion processing and perspective transformation processing of near large and far small, so that while ensuring the generation of the vehicle side view that meets the target acquisition viewing angle, the ground around the vehicle in the generated vehicle side view can be seen on a straight horizon, and the objects around the vehicle in the generated vehicle side view are normal, thereby meeting the intuition of human visual senses and ultimately improving the user experience.

[0077] Based on any of the above embodiments, the multiple coordinates in the unit spherical coordinate system all satisfy the following constraints: ; in,( , , ) is the coordinate in the unit spherical coordinate system; R is the preset radius length.

[0078] Based on any of the above embodiments, the fisheye camera coordinate system and the unit sphere coordinate system have the same origin position; and the coordinate axis directions of the fisheye camera coordinate system and the unit sphere coordinate system are the same.

[0079] Based on any of the above embodiments, the origin position of the virtual camera coordinate system is the same as that of the unit sphere coordinate system.

[0080] Based on any of the above embodiments, the view projection module 330 is specifically used for: Based on the internal parameters of the fisheye camera, converting the multiple coordinates in the virtual camera coordinate system into multiple pixel coordinates corresponding to the fisheye camera; Based on the multiple pixel coordinates, a vehicle side view that meets the target acquisition viewing angle is projected.

[0081] Based on any of the above embodiments, the view projection module 330 is specifically used for: Based on the plurality of pixel coordinates, projecting to obtain an initial image; The initial image is captured based on the target acquisition angle of view to obtain a vehicle side view that meets the target acquisition angle of view.

[0082] The automobile provided by the present invention is described below. The automobile described below and the vehicle side view generating method described above can be referenced to each other.

[0083] The automobile provided by the embodiment of the present invention includes: a fisheye camera and a processor. The fisheye camera is used to collect a fisheye image of the vehicle side of the vehicle. The processor is used to execute a program to implement a vehicle side view generation method as in any of the above embodiments, for example, the method includes: converting multiple coordinates in a fisheye camera coordinate system into multiple coordinates in a unit spherical coordinate system; the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed on a vehicle, and the distances between the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system are the same; based on the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system, converting the multiple coordinates in the unit spherical coordinate system into multiple coordinates in the virtual camera coordinate system; the virtual camera coordinate system is a camera coordinate system corresponding to a virtual camera that meets the target acquisition angle of view; based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, projecting a vehicle side view that meets the target acquisition angle of view.

[0084] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the vehicle side view generation method, which includes: converting multiple coordinates in the fisheye camera coordinate system into multiple coordinates in the unit spherical coordinate system; the fisheye camera coordinate system is a camera coordinate system corresponding to the fisheye camera deployed on the vehicle, and the distances between the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system are the same; based on the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system, converting the multiple coordinates in the unit spherical coordinate system into multiple coordinates in the virtual camera coordinate system; the virtual camera coordinate system is a camera coordinate system corresponding to the virtual camera that meets the target acquisition angle of view; based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, projecting to obtain the vehicle side view that meets the target acquisition angle of view.

[0085] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0086] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the vehicle side view generation method provided by the above-mentioned methods, which includes: converting multiple coordinates in a fisheye camera coordinate system into multiple coordinates in a unit sphere coordinate system; the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed on a vehicle, and the distances between the coordinate points corresponding to the multiple coordinates in the unit sphere coordinate system and the origin of the unit sphere coordinate system are the same; based on the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system, converting the multiple coordinates in the unit sphere coordinate system into multiple coordinates in the virtual camera coordinate system; the virtual camera coordinate system is a camera coordinate system corresponding to a virtual camera that meets the target acquisition viewing angle; based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, projecting a vehicle side view that meets the target acquisition viewing angle.

[0087] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the vehicle side view generation method provided by the above-mentioned methods, the method comprising: converting multiple coordinates in a fisheye camera coordinate system into multiple coordinates in a unit sphere coordinate system; the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed on a vehicle, and the distances between the coordinate points corresponding to the multiple coordinates in the unit sphere coordinate system and the origin of the unit sphere coordinate system are all the same; based on the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system, converting the multiple coordinates in the unit sphere coordinate system into multiple coordinates in the virtual camera coordinate system; the virtual camera coordinate system is a camera coordinate system corresponding to a virtual camera that meets the target acquisition viewing angle; based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, projecting a vehicle side view that meets the target acquisition viewing angle.

[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a vehicle side view, characterized in that: include: Convert multiple coordinates in a fisheye camera coordinate system into multiple coordinates in a unit spherical coordinate system; the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed on a vehicle, and the distances between coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system are all the same; Based on the rotation matrix of the virtual camera coordinate system relative to the unit sphere coordinate system, multiple coordinates in the unit sphere coordinate system are converted into multiple coordinates in the virtual camera coordinate system; the virtual camera coordinate system is a camera coordinate system corresponding to the virtual camera that satisfies the target acquisition viewing angle; Based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, a side view of the vehicle that meets the target acquisition viewing angle is projected.

2. The vehicle side view generation method according to claim 1, characterized in that: The multiple coordinates in the unit spherical coordinate system all satisfy the following constraints: ； in,( , , ) is the coordinate in the unit spherical coordinate system; R is the preset radius length.

3. The vehicle side view generation method according to claim 1 or 2, characterized in that: The fisheye camera coordinate system and the unit sphere coordinate system have the same origin position; and the coordinate axis directions of the fisheye camera coordinate system and the unit sphere coordinate system are the same.

4. The vehicle side view generation method according to claim 1 or 2, characterized in that: The virtual camera coordinate system has the same origin position as the unit sphere coordinate system.

5. The vehicle side view generation method according to claim 1 or 2, characterized in that: The projecting, based on the internal parameters of the fisheye camera and the multiple coordinates in the virtual camera coordinate system, to obtain a side view of the vehicle that satisfies the target acquisition viewing angle includes: Based on the internal parameters of the fisheye camera, converting the multiple coordinates in the virtual camera coordinate system into multiple pixel coordinates corresponding to the fisheye camera; Based on the multiple pixel coordinates, a vehicle side view that meets the target acquisition viewing angle is projected.

6. The vehicle side view generation method according to claim 5, characterized in that: The projecting, based on the multiple pixel coordinates, to obtain a vehicle side view that satisfies the target acquisition viewing angle includes: Based on the plurality of pixel coordinates, projecting to obtain an initial image; The initial image is captured based on the target acquisition angle of view to obtain a vehicle side view that meets the target acquisition angle of view.

7. A vehicle side view generating device, characterized in that: include: A first conversion module is used to convert multiple coordinates in a fisheye camera coordinate system into multiple coordinates in a unit spherical coordinate system; the fisheye camera coordinate system is a camera coordinate system corresponding to a fisheye camera deployed on a vehicle, and the distances between the coordinate points corresponding to the multiple coordinates in the unit spherical coordinate system and the origin of the unit spherical coordinate system are all the same; A second conversion module is used to convert multiple coordinates in the unit spherical coordinate system into multiple coordinates in the virtual camera coordinate system based on the rotation matrix of the virtual camera coordinate system relative to the unit spherical coordinate system; the virtual camera coordinate system is a camera coordinate system corresponding to the virtual camera that meets the target acquisition viewing angle; The view projection module is used to project a side view of the vehicle that meets the target acquisition viewing angle based on the internal parameters of the fisheye camera and multiple coordinates in the virtual camera coordinate system.

8. A vehicle, characterized in that: include: A fisheye camera, the fisheye camera is used to collect a fisheye image of a vehicle side of the vehicle; A processor, wherein the processor is used to execute a program to implement the vehicle side view generation method as described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the vehicle side view generating method according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vehicle side view generating method according to any one of claims 1 to 6 is implemented.

Images