Panoramic image generation method and device, vehicle, storage medium and program product

Abstract

The invention relates to a panoramic image generation method and device, a vehicle, a storage medium and a program product, and the method comprises the steps: obtaining the real-time suspension height of the vehicle in response to the condition that the vehicle has a camera external parameter calibration demand; calibrating external parameters of a camera of the vehicle according to the real-time suspension height to obtain calibrated external parameters of the camera; and generating a panoramic image according to the calibrated external reference of the camera and the image acquired by the camera. According to the technical scheme, the user experience and the driving safety can be improved in an intelligent cabin scene applying the panoramic image.

Description

Technical Field

[0001] The present disclosure relates to the field of visual technology, and in particular to a method, device, vehicle, storage medium, and program product for generating a panoramic image. Background Art

[0002] In smart cockpits, the Around View Monitor (AVM) system is an advanced automotive imaging technology that uses multiple cameras to capture images of the vehicle's surroundings and stitches them together to create a complete 360-degree panoramic image. This system is highly practical in smart cockpits, enhancing user experience and driving safety. The panoramic image can be generated based on camera parameters and the multi-view images captured by the cameras. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides a method, device, vehicle, storage medium and program product for generating a panoramic image.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for generating a panoramic image is provided, comprising: in response to a vehicle having a camera extrinsic parameter calibration requirement, obtaining the real-time suspension height of the vehicle; calibrating the camera extrinsics of the vehicle according to the real-time suspension height to obtain calibrated camera extrinsics; and generating a panoramic image according to the calibrated camera extrinsics and an image captured by the camera.

[0005] When a vehicle requires camera extrinsic calibration, the real-time suspension height of the vehicle is obtained and used to calibrate the vehicle's camera extrinsics to obtain the calibrated camera extrinsics. The calibrated camera extrinsics and the images captured by the camera are then used to generate a panoramic image. Since changes in the vehicle's suspension height will cause changes in the camera extrinsics, if the camera extrinsics are not calibrated in a timely manner when the camera extrinsics change, the panoramic image generation effect will be affected, causing problems such as stitching misalignment in the panoramic image. Therefore, calibrating the camera extrinsics through real-time suspension height does not involve complex image processing processes, is simple and easy to implement, has high real-time performance, and can achieve comprehensive calibration of the camera extrinsics, thereby ensuring the effect of the panoramic image and reducing problems such as stitching misalignment in the panoramic image. Furthermore, in smart cockpit scenarios that use panoramic images, user experience and driving safety can be improved.

[0006] In a possible implementation, the method for generating a panoramic image further includes: acquiring suspension height change information of the vehicle; and determining, based on the suspension height change information, whether the vehicle has a camera extrinsic parameter calibration requirement.

[0007] Changes in the vehicle's suspension height cause changes in the camera's external parameters (i.e., camera extrinsics). This change is reflected in varying degrees of suspension height. Therefore, using this information to determine whether the vehicle requires camera extrinsic calibration, the system can adapt in real time to various scenarios where these changes can cause changes in camera extrinsics, resolving the issue of poor panoramic image quality in these scenarios.

[0008] In one possible implementation, the suspension height change information includes a suspension height change value, and determining that the vehicle has a camera extrinsic parameter calibration requirement based on the suspension height change information includes: when the suspension height change value is higher than a preset suspension height change value, determining that the vehicle has a camera extrinsic parameter calibration requirement.

[0009] When the suspension height change value is higher than the preset suspension height change value, the camera extrinsic parameter calibration is triggered so that the camera extrinsic parameter calibration can dynamically adapt to the change of the suspension height.

[0010] In one possible embodiment, obtaining the suspension height change information of the vehicle includes: obtaining the real-time suspension heights respectively detected by multiple suspension height detection devices of the vehicle; obtaining the historical suspension heights respectively detected by the multiple suspension height detection devices, the historical suspension heights being the suspension heights detected at the time of the previous camera extrinsic parameter calibration; determining the suspension height change values ​​respectively detected by the multiple suspension height detection devices based on the real-time suspension heights and historical suspension heights respectively detected by the multiple suspension height detection devices; determining the suspension height change information based on the suspension height change values ​​respectively detected by the multiple suspension height detection devices.

[0011] By combining the real-time detection data and historical detection data of multiple suspension height detection devices to determine the suspension height change information, accurate determination of the height change information can be achieved.

[0012] In a possible implementation, the method for generating a panoramic image further includes: acquiring vehicle operation information related to a change in suspension height of the vehicle; and determining, based on the vehicle operation information, whether the vehicle has a camera extrinsic parameter calibration requirement.

[0013] By using vehicle operation information related to the change in suspension height of the vehicle, it is determined whether the vehicle has the need for camera extrinsic parameter calibration. This is equivalent to indirectly determining whether the vehicle has the need for camera extrinsic parameter calibration based on the change in suspension height. As a result, it can adapt in real time to various scenarios where camera extrinsic parameters change due to changes in suspension height, and solve the problem of poor panoramic image effects in these scenarios.

[0014] In one possible embodiment, the vehicle detects the suspension height through a suspension height detection device, and calibrates the camera extrinsics of the vehicle according to the real-time suspension height to obtain the calibrated camera extrinsics, including: obtaining the preset position coordinates of the suspension height detection device; determining the real-time position coordinates of the suspension height detection device according to the real-time suspension height and the preset position coordinates; determining the posture transformation matrix of the vehicle according to the preset position coordinates and the real-time position coordinates; and calibrating the camera extrinsics of the vehicle according to the posture transformation matrix to obtain the calibrated camera extrinsics.

[0015] Because the suspension height detection device is part of the vehicle body, its shape remains fixed regardless of its turbulence. Therefore, the motion of the suspension height detection device is a rigid body motion. Any complex motion of a rigid body can be decomposed into a combination of translation and rotation. Based on this principle, the position coordinates of the suspension height detection device can be analyzed and calculated to obtain a pose transformation matrix. This pose transformation matrix represents the transformation of the vehicle's current pose compared to its original pose. Therefore, using the pose transformation matrix, the vehicle's camera extrinsic parameters can be effectively, simply, and accurately calibrated.

[0016] In a possible implementation, calibrating the vehicle's camera extrinsic parameters based on the pose transformation matrix to obtain calibrated camera extrinsic parameters includes: decomposing the pose transformation matrix to obtain a rotation matrix and a translation matrix; and calibrating the vehicle's camera extrinsic parameters based on the rotation matrix and the translation matrix to obtain calibrated camera extrinsic parameters.

[0017] Since camera extrinsic parameters usually involve rotation parameters and translation parameters, by decomposing the pose transformation matrix, a simple, effective, accurate and comprehensive calibration of the camera extrinsic parameters can be achieved based on the rotation matrix and translation matrix obtained after the decomposition.

[0018] In one possible implementation, calibrating the camera extrinsics of the vehicle according to the rotation matrix and the translation matrix to obtain calibrated camera extrinsics includes: calibrating the rotation parameters included in the camera extrinsics of the vehicle according to the rotation matrix to obtain calibrated rotation parameters; calibrating the translation parameters included in the camera extrinsics of the vehicle according to the rotation matrix and the translation parameters to obtain calibrated translation parameters; and determining the calibrated camera extrinsics based on the calibrated rotation parameters and the calibrated translation parameters.

[0019] The rotation parameters are calibrated through the rotation matrix, and the translation parameters are calibrated through the rotation matrix and translation parameters. In this way, a comprehensive calibration of the camera extrinsic parameters can be achieved.

[0020] In a possible implementation manner, the preset position coordinates of the suspension height detection device are position coordinates calibrated during the factory production stage of the vehicle.

[0021] During the factory production stage of the vehicle, the state of the vehicle is fixed. Calibration is performed in this fixed state to obtain preset position coordinates. The preset position coordinates can be used as a reference position coordinate to determine the posture transformation matrix, thereby reducing the difficulty of determining the posture transformation matrix.

[0022] In one possible implementation, the calibrated camera extrinsic parameters include calibration extrinsic parameters corresponding to each of the multiple cameras of the vehicle. Generating a panoramic image based on the calibrated camera extrinsic parameters and images captured by the cameras includes: capturing images of the vehicle's surroundings at different perspectives using the multiple cameras to obtain multiple images; and stitching the multiple images together based on the calibration extrinsic parameters corresponding to the multiple cameras to obtain a panoramic image.

[0023] Multiple cameras capture images of the vehicle's surroundings from different perspectives, generating multiple images corresponding to different perspectives. Furthermore, by stitching these images together using the calibration parameters corresponding to each camera, a better panoramic image can be obtained.

[0024] According to a second aspect of an embodiment of the present disclosure, a device for generating a panoramic image is provided, which is configured to execute the method for generating a panoramic image as described in the first aspect of the present disclosure.

[0025] According to a third aspect of an embodiment of the present disclosure, a vehicle is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to: execute the executable instructions to implement the method for generating a panoramic image as described in the first aspect of the present disclosure.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the method for generating a panoramic image described in the first aspect of the present disclosure is implemented.

[0027] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method for generating a panoramic image as described in the first aspect of the present disclosure.

[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0030] Figure 1 The figure is a flowchart of a method for generating a panoramic image according to an exemplary embodiment.

[0031] Figure 2 is a schematic diagram of a reference coordinate system of a vehicle according to an exemplary embodiment.

[0032] Figure 3 The figure is a flowchart of an application of camera extrinsic calibration according to an exemplary embodiment.

[0033] Figure 4 FIG. 1 is a schematic diagram of a panoramic image according to a related technology according to an exemplary embodiment.

[0034] Figure 5 The diagram is a schematic diagram of a panoramic image obtained by using the technical solution of an embodiment of the present disclosure, according to an exemplary embodiment.

[0035] Figure 6 The present invention is a block diagram of a device for generating a panoramic image according to an exemplary embodiment.

[0036] Figure 7 is a block diagram of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0037] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0038] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the corresponding data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0039] In smart cockpits, surround view systems are an advanced automotive imaging technology that uses multiple cameras to capture images of the vehicle's surroundings and stitch them together to create a complete 360-degree panoramic view. These systems are highly practical in smart cockpits, enhancing user experience and driving safety.

[0040] The implementation process of a surround view system involves: First, the vehicle's camera's intrinsic and extrinsic parameters are preset, for example, through factory calibration. Then, based on these intrinsic and extrinsic parameters, images captured from different perspectives are processed in real time, including correction, stitching, and rendering, ultimately presenting a complete 360-degree panoramic image. The camera's extrinsic parameters are related to the camera's relative position and attitude.

[0041] However, due to the differences between the various driving environments of vehicles and the environments in which the vehicle's preset camera extrinsics are set, the camera extrinsics will change accordingly. If the preset camera extrinsics are still used to generate panoramic images, problems such as panoramic image stitching errors will occur. Therefore, it is necessary to calibrate the preset camera extrinsics to ensure the presentation effect of the panoramic image.

[0042] In related technologies, the vehicle is placed in a specific scene, and then camera images of different perspectives are matched to adjust some parameters in the camera extrinsic parameters to achieve camera extrinsic parameter calibration. This camera extrinsic parameter calibration method requires the vehicle to be in a specific scene, such as with relatively clear white lane lines on both sides of the vehicle body to facilitate feature extraction. In addition, this camera extrinsic parameter calibration method involves relatively complex processes such as image processing and feature extraction, which requires high computing power, is time-consuming, and has poor real-time performance. In addition, due to image feature limitations, usually only some of the camera extrinsic parameters can be adjusted. However, changes in camera extrinsic parameters usually involve changes in more than just some of the extrinsic parameters.

[0043] Therefore, the camera extrinsic parameter calibration method of the related art has problems such as complex calibration process, poor real-time performance, and poor calibration effect.

[0044] Considering that the change of vehicle suspension height will cause the change of camera extrinsic parameters, if the camera extrinsic parameters are not calibrated in time when the camera extrinsic parameters change, the effect of the panoramic image will be affected, causing problems such as stitching misalignment in the panoramic image.

[0045] Based on this, the disclosed embodiments provide a technical solution that calibrates camera extrinsics using real-time suspension height. This solution, which does not involve complex image processing, is simple and easy to implement, with high real-time performance. Furthermore, it can achieve comprehensive calibration of camera extrinsics, thereby ensuring panoramic image quality and reducing issues such as stitching misalignment. Furthermore, in smart cockpit scenarios using panoramic images, this solution can enhance user experience and driving safety.

[0046] The technical solutions provided by the embodiments of the present disclosure can be applied to various scenarios such as panoramic surround view, assisted driving, computer vision, camera calibration, etc.

[0047] Figure 1 is a flow chart showing a method for generating a panoramic image according to an exemplary embodiment. Figure 1As shown, the method for generating a panoramic image can be applied to a vehicle, and the method for generating a panoramic image includes the following steps: Step S11: In response to the vehicle having a camera extrinsic calibration requirement, the real-time suspension height of the vehicle is obtained.

[0048] Step S12: calibrate the vehicle's camera extrinsic parameters according to the real-time suspension height to obtain calibrated camera extrinsic parameters.

[0049] Step S13: Process the image captured by the camera according to the calibrated camera extrinsic parameters to obtain a panoramic image.

[0050] In steps S11 to S13, when the vehicle has a camera extrinsic parameter calibration requirement, the camera extrinsic parameters are calibrated using the real-time suspension height, and then the calibrated camera extrinsic parameters are used to process the image captured by the camera to obtain a panoramic image.

[0051] It is understandable that, when necessary, calibration of camera extrinsic parameters can also reduce unnecessary performance consumption and improve vehicle performance.

[0052] Whether a vehicle requires camera extrinsic calibration can be determined in a variety of ways. Some possible implementations are described below.

[0053] In a possible implementation, determining whether the vehicle has a camera extrinsic parameter calibration requirement may include: acquiring suspension height change information of the vehicle; and determining, based on the suspension height change information, that the vehicle has a camera extrinsic parameter calibration requirement.

[0054] It's understandable that changes in a vehicle's suspension height can cause changes in the camera's external parameters (i.e., camera extrinsics). This change is reflected in varying degrees of suspension height. Therefore, using this information to determine whether a vehicle requires camera extrinsic calibration, the system can adapt in real time to various scenarios where camera extrinsics change due to suspension height changes, resolving the issue of poor panoramic image quality in these scenarios.

[0055] Moreover, the related technologies rely on specific vehicle surrounding scenes to trigger camera extrinsic parameter calibration, while the technical solution of the disclosed embodiment can trigger the camera extrinsic parameter calibration requirements at any time according to the suspension height change information, thereby improving real-time performance.

[0056] In some embodiments, the vehicle's suspension height may change due to various reasons. If the camera's external parameters are not updated with the suspension height, the 360-degree panoramic image may be misaligned, affecting the user experience and potentially causing safety hazards.

[0057] For example, when the vehicle load changes, the vehicle's suspension height will change; when the vehicle is driving on a bumpy road, the suspension height will change due to the bumps in the vehicle body; some vehicles support manual adjustment of the suspension height, and users can adjust the suspension height as needed, and the suspension height can change due to manual adjustment.

[0058] In some embodiments, the suspension height change information may include a suspension height change value, which may be the difference between the real-time suspension height and the historical suspension height. The historical suspension height may be the suspension height at the time of the previous camera extrinsic parameter calibration.

[0059] Based on the suspension height change value and according to the suspension height change information, determining that the vehicle has a camera extrinsic parameter calibration requirement includes: when the suspension height change value is higher than a preset suspension height change value, determining that the vehicle has a camera extrinsic parameter calibration requirement.

[0060] In this embodiment, when the suspension height change value is higher than a preset suspension height change value, the camera extrinsic parameter calibration is triggered, so that the camera extrinsic parameter calibration can dynamically adapt to the change of the suspension height.

[0061] In some embodiments, the preset suspension height change value can be set based on the vehicle's own conditions, the performance and effect of the camera extrinsic parameter calibration algorithm, etc. Specifically, the appropriate suspension height change value can be set through preliminary actual measurement, simulation testing, etc.

[0062] As an example, the preset suspension height change value may be within the range of 3 mm to 4 mm.

[0063] In some embodiments, the suspension height may be detected by a suspension height detection device, and thus the suspension height change information may be determined.

[0064] As an example, the suspension height detection device may be a suspension height sensor, which may be located at the vehicle eyebrow above the wheel.

[0065] In some embodiments, a vehicle may be equipped with multiple suspension height sensors. As an example, a vehicle may be equipped with four suspension height sensors, each located on the brow of a different wheel. For example, a suspension height sensor A may be located above the left front wheel, a suspension height sensor B may be located above the right front wheel, a suspension height sensor C may be located above the left rear wheel, and a suspension height sensor D may be located above the right rear wheel.

[0066] In the case where a vehicle is provided with a plurality of suspension height detection devices, the suspension height change information may be determined by combining detection data from the plurality of suspension height detection devices.

[0067] Therefore, as an optional implementation, obtaining the suspension height change information of the vehicle includes: obtaining the real-time suspension heights detected by multiple suspension height detection devices of the vehicle; obtaining the historical suspension heights detected by multiple suspension height detection devices, the historical suspension heights being the suspension heights detected at the time of the previous camera external parameter calibration; determining the suspension height change values ​​detected by multiple suspension height detection devices based on the real-time suspension heights and historical suspension heights detected by multiple suspension height detection devices; determining the suspension height change information based on the suspension height change values ​​detected by multiple suspension height detection devices.

[0068] In this embodiment, the real-time detection data and historical detection data of a plurality of suspension height detection devices are combined to determine the suspension height change information, thereby achieving accurate determination of the height change information.

[0069] Taking multiple suspension height detection devices including suspension height sensor A, suspension height sensor B, suspension height sensor C and suspension height sensor D as an example, the historical suspension heights detected by the four suspension height sensors are: heightA new 、heightB new 、heightC new 、heightD new ; The historical suspension heights detected by the four suspension height sensors are: heightA old 、heightB old 、heightC old 、heightD old .

[0070] Then, the suspension height change values ​​detected by the four suspension height sensors are: |heightA new - heightA old |、|heightB new - heightB old |、|heightC new - heightC old |、|heightD new - heightD old |.

[0071] Accordingly, the suspension height change value included in the suspension height change information may be the sum of the suspension height change values ​​respectively detected by the four suspension height sensors.

[0072] Therefore, the suspension height change value deltaH can be: | heightA new - heightA old |+|heightBnew -heightB old |+|heightC new - heightC old |+|heightD new - heightD old |.

[0073] In some embodiments, the number of suspension height sensors may be greater than or equal to 3. When the number of suspension height sensors is other embodiments, the corresponding suspension height change value may be determined by referring to the embodiments herein.

[0074] In some embodiments, when the suspension height change value is lower than a preset suspension height change value, it is determined that the vehicle does not require camera extrinsic calibration.

[0075] It is understandable that the change in suspension height may be caused by other reasons. Therefore, in addition to directly detecting the suspension height change information, it is also possible to determine whether the vehicle has a camera extrinsic parameter calibration requirement through some related information.

[0076] Therefore, as an optional implementation, determining whether the vehicle has a camera extrinsic parameter calibration requirement includes: obtaining vehicle operation information related to the change in the vehicle's suspension height; and determining, based on the vehicle operation information, whether the vehicle has a camera extrinsic parameter calibration requirement.

[0077] In this embodiment, whether the vehicle has a camera extrinsic parameter calibration requirement is determined by using vehicle operation information related to the suspension height change of the vehicle, which is equivalent to indirectly determining whether the vehicle has a camera extrinsic parameter calibration requirement based on the suspension height change. Therefore, it can adapt to various scenarios of camera extrinsic parameter changes caused by suspension height changes in real time, and solve the problem of poor panoramic image effect in these scenarios.

[0078] In some embodiments, the vehicle operation information related to the change in the vehicle's suspension height includes, for example, the vehicle's load, the vehicle's driving mode, and the vehicle's driving environment.

[0079] As an example, if the vehicle's load varies significantly, it is determined that the vehicle requires camera extrinsic calibration. In this case, the suspension height will change accordingly, and the camera extrinsic calibration can be performed.

[0080] For example, if the vehicle's driving mode is Sport mode, it is determined that the vehicle requires camera extrinsic calibration. In this case, the vehicle is traveling at a high speed, which may cause the vehicle to be bumpy, resulting in changes in suspension height. Therefore, camera extrinsic calibration may be required.

[0081] As an example, if the vehicle is traveling on a bumpy road, it is determined that the vehicle requires camera extrinsic calibration. In this case, the suspension height changes due to the bumpy vehicle body, and the camera extrinsic calibration can be performed.

[0082] In some embodiments, if the vehicle operating information does not meet the corresponding conditions, it can be determined that the vehicle does not require camera extrinsic calibration. For example, if the vehicle load changes slightly or the vehicle is traveling on a flat road, it can be determined that the vehicle does not require camera extrinsic calibration.

[0083] It is understandable that in the application scenario of the smart cockpit, the vehicle and the computer are highly interactive. Therefore, it is also possible to determine whether the vehicle has camera extrinsic parameter calibration requirements based on user requests.

[0084] Therefore, as an optional implementation, determining that the vehicle has a camera extrinsic parameter calibration requirement includes: in response to receiving a user instruction related to camera extrinsic parameter calibration, determining that the vehicle has a camera extrinsic parameter calibration requirement.

[0085] In some embodiments, the user instruction related to camera extrinsic calibration may indicate that there is stitching misalignment in the panoramic image; or, it may indicate that the user has manually adjusted the suspension height, etc., which is not limited here.

[0086] In some embodiments, the real-time suspension height of the vehicle is the real-time suspension height detected by the suspension height detection device. Therefore, the real-time suspension height can be directly obtained from the suspension height detection device.

[0087] Also, referring to the implementation of the aforementioned embodiment, the vehicle may be equipped with a plurality of suspension height detection devices, and the real-time suspension height includes the real-time suspension heights respectively detected by the plurality of suspension height detection devices.

[0088] As an optional implementation, step S12 includes: obtaining the preset position coordinates of the suspension height detection device; determining the real-time position coordinates of the suspension height detection device based on the real-time suspension height and the preset position coordinates; determining the vehicle's posture transformation matrix based on the preset position coordinates and the real-time position coordinates; calibrating the vehicle's camera extrinsic parameters based on the posture transformation matrix to obtain the calibrated camera extrinsic parameters.

[0089] Because the suspension height detection device is part of the vehicle body, its shape remains fixed regardless of its turbulence. Therefore, the motion of the suspension height detection device is a rigid body motion. Any complex motion of a rigid body can be decomposed into a combination of translation and rotation. Based on this principle, the position coordinates of the suspension height detection device can be analyzed and calculated to obtain a pose transformation matrix. This pose transformation matrix represents the transformation of the vehicle's current pose compared to its original pose. Therefore, using the pose transformation matrix, the vehicle's camera extrinsic parameters can be effectively, simply, and accurately calibrated.

[0090] In some embodiments, the preset position coordinates of the suspension height detection device are position coordinates calibrated during the factory production stage of the vehicle.

[0091] It can be understood that during the factory production stage of the vehicle, the state of the vehicle is fixed. By calibrating in this fixed state, the preset position coordinates can be obtained. The preset position coordinates can be used as a reference position coordinate to determine the posture transformation matrix, thereby reducing the difficulty of determining the posture transformation matrix.

[0092] In some embodiments, the preset position coordinates may be position coordinates in a reference coordinate system of the vehicle, and the position coordinates may be 3D coordinates. Furthermore, the preset position coordinates may include 3D coordinates corresponding to a plurality of suspension height detection devices.

[0093] Figure 2 is a schematic diagram showing a reference coordinate system of a vehicle according to an exemplary embodiment. Figure 2 As shown, in this reference coordinate system, the X-axis points from the left side of the vehicle to the right side of the vehicle, the Y-axis points from the rear of the vehicle to the front of the vehicle, the Z-axis points vertically above the vehicle (not shown in the figure), and the coordinate origin is the projection position of the vehicle center on the ground.

[0094] Taking four suspension height sensors as an example, the preset position coordinates may include 3D coordinates corresponding to the four suspension height sensors respectively.

[0095] For example, the preset 3D coordinates of suspension height sensor A to suspension height sensor D may be: posA base =(xA base , yA base , zA base ) posB base =(xB base , yB base , zB base ) posC base =(xC base , yC base , zCbase ) posD base =(xD base ,yD base , zD base ) Regarding the specific calibration method of coordinates, you can refer to the mature technology in this field and will not introduce it in detail here.

[0096] In some embodiments, the real-time position coordinates of the suspension height detection device may be determined based on the real-time suspension height and the preset position coordinates.

[0097] In some embodiments, the real-time position coordinates of the suspension height detection device in the vehicle's reference coordinate system can be calculated by combining the real-time suspension height and preset position coordinates through geometric analysis and rigid body kinematics. The real-time suspension height can be used to determine the real-time z-axis coordinate of the suspension height detection device.

[0098] Continuing with the example of four suspension height sensors, the preset position coordinates of the four suspension height sensors are: (posA base ,posB base ,posC base ,posD base ), the real-time suspension height detected by the four suspension height sensors: (heightA new 、heightB new 、heightC new 、heightD new ), through geometric analysis and rigid body kinematics calculation analysis, the real-time position coordinates of the four suspension height sensors are obtained: posA new =(xA new , yA new , zA new ) posB new =(xB new , yB new , zB new ) posC new =(xC new , yC new , zC new ) posD new =(xD new ,yD new , zD new ) Among them, zA new According to zA base and heightA newCalculated, zB new According to zB base and heightB new Calculated, zC new According to zC base and heightC new Calculated, zD new According to zD base and heightD new Calculated.

[0099] In some embodiments, the relationship between the preset position coordinates, the real-time position coordinates and the pose transformation matrix can be: Mat×(posA base ,posB base ,posC base ,posD base ) = (posA new ,posB new ,posC new ,posD new ), where Mat represents the pose transformation matrix.

[0100] Therefore, the pose transformation matrix can be expressed as: Mat = (posA base ,posB base ,posC base ,posD base )×(posA new ,posB new ,posC new ,posD new ) -1 .

[0101] In some embodiments, the vehicle's camera extrinsic parameters are calibrated according to the posture transformation matrix to obtain the calibrated camera extrinsic parameters, including: decomposing the posture transformation matrix to obtain a rotation matrix and a translation matrix; calibrating the vehicle's camera extrinsic parameters according to the rotation matrix and the translation matrix to obtain the calibrated camera extrinsic parameters.

[0102] Since camera extrinsic parameters usually involve rotation parameters and translation parameters, by decomposing the pose transformation matrix, a simple, effective, accurate and comprehensive calibration of the camera extrinsic parameters can be achieved based on the rotation matrix and translation matrix obtained after the decomposition.

[0103] As an example, , where R represents the rotation matrix and T represents the translation matrix.

[0104] In some embodiments, the vehicle's camera extrinsics are calibrated according to the rotation matrix and the translation matrix to obtain calibrated camera extrinsics, including: calibrating the rotation parameters included in the vehicle's camera extrinsics according to the rotation matrix to obtain calibrated rotation parameters; calibrating the translation parameters included in the vehicle's camera extrinsics according to the rotation matrix and the translation parameters to obtain calibrated translation parameters; and determining the calibrated camera extrinsics based on the calibrated rotation parameters and the calibrated translation parameters.

[0105] In this embodiment, the rotation parameters are calibrated by the rotation matrix, and the translation parameters are calibrated by the rotation matrix and the translation parameters, thereby achieving a comprehensive calibration of the camera extrinsic parameters.

[0106] As an example, the camera extrinsics to be calibrated are expressed as: , where R base is the rotation parameter to be calibrated (usually a 3×3 rotation matrix), T base is the translation parameter to be calibrated (usually a 3×1 translation matrix).

[0107] Then, the extrinsic parameters of the calibrated camera can be expressed as: .

[0108] Therefore, the rotation parameter after calibration is: R new =R base ×R, calibrated translation parameter: T new =T base +R base ×T.

[0109] It can be seen that the calibrated rotation parameter is the product of the rotation parameter to be calibrated and the rotation matrix, and the calibrated translation parameter is the parameter obtained by adding the translation parameter to be calibrated to the product of the rotation parameter to be calibrated and the translation matrix.

[0110] In some embodiments, after completing the calibration of the camera extrinsic parameters, the real-time suspension height can be saved as the suspension height at the current camera extrinsic parameter calibration moment. Thus, the next time when determining whether there is a need for camera extrinsic parameter calibration, the suspension height change information can be determined using the suspension height.

[0111] It can be seen that the above-mentioned camera extrinsic parameter calibration method does not involve complex image processing procedures. It calibrates the camera extrinsic parameters through real-time suspension height, which is easy to implement in engineering.

[0112] Furthermore, after obtaining the calibrated camera extrinsic parameters, in step S13, the image captured by the camera is processed according to the calibrated camera extrinsic parameters to obtain a panoramic image.

[0113] In some embodiments, the vehicle includes multiple cameras, which respectively capture images of the vehicle's surroundings at different viewing angles. A panoramic image can be obtained by stitching the images captured by the multiple cameras.

[0114] Therefore, during the camera extrinsic parameter calibration process, the extrinsic parameters of the multiple cameras need to be calibrated separately, and the extrinsic parameters of each camera can be calibrated in the same way.

[0115] Therefore, the calibrated camera extrinsic parameters include the calibration extrinsic parameters corresponding to the multiple cameras of the vehicle respectively.

[0116] Furthermore, step S13 may include: acquiring images of the surrounding environment of the vehicle at different viewing angles using multiple cameras to obtain multiple images; and stitching the multiple images according to calibration extrinsic parameters corresponding to the multiple cameras to obtain a panoramic image.

[0117] In this embodiment, multiple cameras capture images of the vehicle's surroundings from different perspectives, resulting in multiple images corresponding to different perspectives. Furthermore, by stitching these images together using the calibration extrinsic parameters corresponding to each of the multiple cameras, a more effective panoramic image can be obtained.

[0118] In some embodiments, in addition to using camera extrinsic parameters, more information related to image stitching, such as camera intrinsic parameters, can also be combined to achieve image stitching and obtain a panoramic image. Specifically, reference can be made to the mature panoramic image generation technology in the field, which will not be introduced in detail here.

[0119] In some embodiments, the generation of a panoramic image may involve not only image stitching but also image correction, image enhancement, image denoising, etc., which are not limited here.

[0120] In some embodiments, the obtained panoramic image can be used for smart cockpit functions such as assisted driving and high-precision map display, which is not limited here.

[0121] Figure 3 FIG. 1 is an application flow chart of camera extrinsic calibration according to an exemplary embodiment. Figure 3 As shown, the application process includes: First, obtain the 3D coordinates of the suspension height sensor at the factory calibration time.

[0122] Next, during use of the vehicle, a suspension height value output by the suspension height sensor at the current moment is obtained.

[0123] Then, a change value of the suspension height value output by the suspension height sensor at the current moment compared with the suspension height at the last external parameter calibration moment is calculated.

[0124] Furthermore, it is determined whether the change value is greater than a set threshold. If not, the steps of obtaining the change value and determining can be repeated.

[0125] If yes, the 3D coordinates of the suspension height sensor at the current moment can be calculated based on rigid body kinematics analysis.

[0126] Furthermore, a posture transformation matrix is ​​calculated based on the 3D coordinates of the suspension height sensor at the factory calibration time and the 3D coordinates of the suspension height sensor at the current time.

[0127] Next, the pose transformation matrix is ​​decomposed into a rotation matrix and a translation matrix.

[0128] Then, the camera extrinsics are calibrated using the rotation matrix and translation matrix.

[0129] The technical solution of the embodiments of the present disclosure can solve the problem of stitching misalignment in panoramic images in scenarios with high suspension heights. In scenarios with standard or lower suspension heights, the possibility of stitching misalignment in panoramic images is low.

[0130] Regarding the higher suspension height, as an example, if the suspension heights of the left front wheel, right front wheel, left rear wheel, and right rear wheel are 47mm, 48mm, 50mm, and 48mm respectively, the suspension height can be considered to be higher.

[0131] Figure 4 is a schematic diagram of a panoramic image of a related technology according to an exemplary embodiment. Figure 4 As shown in the figure, since the relevant technology does not adopt a reasonable camera extrinsic parameter calibration method when the suspension height changes, the spliced ​​part of the lane line is misaligned.

[0132] Figure 5 FIG. 1 is a schematic diagram of a panoramic image obtained by using the technical solution of an embodiment of the present disclosure according to an exemplary embodiment. Figure 5 As shown, by adopting the camera extrinsic parameter calibration method of the embodiment of the present disclosure, the obtained panoramic image does not have the lane line misalignment problem, thereby ensuring the stitching effect of the panoramic image.

[0133] Furthermore, the technical solution of the disclosed embodiment can enable the panoramic image presentation effect to adapt in real time to the changes in suspension height under different loads and road conditions, fully ensuring the panoramic image presentation effect, and improving the driver's safety and driving experience in various application scenarios of the smart cockpit.

[0134] Figure 6 FIG. 6 is a block diagram of a device 600 for generating a panoramic image according to an exemplary embodiment. Figure 6 , the device comprises: The acquisition module 601 is configured to: in response to a vehicle having a camera extrinsic parameter calibration requirement, acquire the real-time suspension height of the vehicle.

[0135] The calibration module 602 is configured to calibrate the camera extrinsic parameters of the vehicle according to the real-time suspension height to obtain calibrated camera extrinsic parameters.

[0136] The generating module 603 is configured to generate a panoramic image according to the calibrated camera extrinsic parameters and the image captured by the camera.

[0137] Optionally, the device includes a determination module configured to: obtain suspension height change information of the vehicle; and determine, based on the suspension height change information, whether the vehicle has a camera extrinsic parameter calibration requirement.

[0138] Optionally, the determination module is further configured to: when the suspension height change value is higher than a preset suspension height change value, determine that the vehicle has a camera extrinsic parameter calibration requirement.

[0139] Optionally, the determination module is further configured to: obtain the real-time suspension heights detected by multiple suspension height detection devices of the vehicle; obtain the historical suspension heights detected by the multiple suspension height detection devices, the historical suspension heights being the suspension heights detected at the time of the previous camera extrinsic parameter calibration; determine the suspension height change values ​​detected by the multiple suspension height detection devices based on the real-time suspension heights and historical suspension heights detected by the multiple suspension height detection devices; determine the suspension height change information based on the suspension height change values ​​detected by the multiple suspension height detection devices.

[0140] Optionally, the determination module is further configured to: obtain vehicle operation information related to the change in suspension height of the vehicle; and determine, based on the vehicle operation information, whether the vehicle has a camera extrinsic parameter calibration requirement.

[0141] Optionally, the calibration module 602 is further configured to: obtain the preset position coordinates of the suspension height detection device; determine the real-time position coordinates of the suspension height detection device based on the real-time suspension height and the preset position coordinates; determine the posture transformation matrix of the vehicle based on the preset position coordinates and the real-time position coordinates; calibrate the camera extrinsic parameters of the vehicle based on the posture transformation matrix to obtain the calibrated camera extrinsic parameters.

[0142] Optionally, the calibration module 602 is further configured to: decompose the posture transformation matrix to obtain a rotation matrix and a translation matrix; calibrate the camera extrinsic parameters of the vehicle according to the rotation matrix and the translation matrix to obtain calibrated camera extrinsic parameters.

[0143] Optionally, the calibration module 602 is further configured to: calibrate the rotation parameters included in the camera extrinsic parameters of the vehicle according to the rotation matrix to obtain calibrated rotation parameters; calibrate the translation parameters included in the camera extrinsic parameters of the vehicle according to the rotation matrix and the translation parameters to obtain calibrated translation parameters; determine the calibrated camera extrinsic parameters based on the calibrated rotation parameters and the calibrated translation parameters.

[0144] Optionally, the generation module 603 is further configured to: acquire images of the surrounding environment of the vehicle at different perspectives through the multiple cameras to obtain multiple images; and stitch the multiple images according to the calibration external parameters corresponding to the multiple cameras to obtain a panoramic image.

[0145] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0146] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon. When the program instructions are executed by a processor, the steps of the method for generating a panoramic image provided by the present disclosure are implemented.

[0147] Figure 7 FIG2 is a block diagram illustrating a vehicle 700 according to an exemplary embodiment. For example, vehicle 700 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or another type of vehicle. Vehicle 700 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0148] Reference Figure 7 Vehicle 700 may include various subsystems, such as an infotainment system 710, a perception system 720, a decision-making control system 730, a drive system 740, and a computing platform 750. Vehicle 700 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of vehicle 700 may be interconnected via wired or wireless means.

[0149] In some embodiments, the infotainment system 710 may include a communication system, an entertainment system, a navigation system, and the like.

[0150] Perception system 720 may include several sensors for sensing information about the environment surrounding vehicle 700. For example, perception system 720 may include a global positioning system (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera.

[0151] The decision control system 730 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0152] The drive system 740 may include components that provide power to the vehicle 700. In one embodiment, the drive system 740 may include an engine, a power source, a transmission system, and wheels. The engine may be an internal combustion engine, an electric motor, an air compression engine, or a combination thereof. The engine is capable of converting energy provided by the power source into mechanical energy.

[0153] Some or all functions of the vehicle 700 are controlled by a computing platform 750. The computing platform 750 may include at least one processor 751 and a memory 752. The processor 751 may execute instructions 753 stored in the memory 752.

[0154] The processor 751 can be any conventional processor, such as a commercially available CPU. The processor can also include a graphics processor (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0155] The memory 752 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0156] In addition to instructions 753 , memory 752 may also store data, such as road maps, route information, and vehicle location, direction, speed, etc. The data stored in memory 752 may be used by computing platform 750 .

[0157] In the embodiment of the present disclosure, the processor 751 may execute the instruction 753 to complete all or part of the steps of the above-mentioned method for generating a panoramic image.

[0158] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and has a code portion for executing the above-mentioned method for generating a panoramic image when executed by the programmable device.

[0159] Furthermore, the word "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the use of the word exemplary is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X applies to A or B" is intended to mean any of the natural inclusive permutations. That is, if X applies to A; X applies to B; or X applies to both A and B, then "X applies to A or B" satisfies any of the aforementioned instances. Furthermore, the articles "a" and "an," as used in this application and the appended claims, are generally understood to mean "one or more," unless otherwise specified or clear from the context to refer to the singular form.

[0160] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. With particular regard to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if not structurally equivalent to the disclosed structure. In addition, although particular features of the present disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "include," "have," "have," "have," or variations thereof are used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term "comprising."

[0161] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

[0162] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0163] It should be understood that, unless otherwise specifically noted, the features of the various embodiments of the present disclosure described herein may be combined with each other. As used herein, the term "and / or" includes any one of the relevant listed items and any combination of any two or more thereof; similarly, "at least one of" includes any one of the relevant listed items and any combination of any two or more thereof.

[0164] Although terms such as "first", "second" and "third" may be used herein to describe various components, parts, regions, layers or sections, these components, parts, regions, layers or sections are not limited to these terms. On the contrary, these terms are only used to distinguish one component, part, region, layer or section from another component, part, region, layer or section. Therefore, without departing from the teachings of each example, the first component, part, region, layer or section mentioned in the examples described herein may also be referred to as the second component, part, region, layer or section. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one such feature. In the description herein, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise clearly and specifically defined.

[0165] It should be understood that spatially relative terms, such as "above," "upper," "below," and "lower," are used herein to describe the relationship of one element to another element shown in the figures. Such spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, an element described as being "above" or "upper" relative to another element would then be "below" or "lower" relative to the other element. Thus, the term "above" encompasses both above and below orientations, depending on the spatial orientation of the device. The device may be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and the spatially relative terms used herein should be interpreted accordingly.

Claims

1. A method for generating a panoramic image, characterized in that: include: In response to a vehicle having a camera extrinsic parameter calibration requirement, obtaining a real-time suspension height of the vehicle; Calibrate the camera extrinsic parameters of the vehicle according to the real-time suspension height to obtain calibrated camera extrinsic parameters; A panoramic image is generated based on the calibrated camera extrinsics and the image captured by the camera.

2. The method for generating a panoramic image according to claim 1, wherein: The method for generating a panoramic image further includes: Obtaining suspension height change information of the vehicle; According to the suspension height change information, it is determined that the vehicle has a camera extrinsic parameter calibration requirement.

3. The method for generating a panoramic image according to claim 2, wherein: The suspension height change information includes a suspension height change value, and determining, based on the suspension height change information, whether the vehicle has a camera extrinsic parameter calibration requirement includes: When the suspension height change value is higher than a preset suspension height change value, it is determined that the vehicle has a camera extrinsic parameter calibration requirement.

4. The method for generating a panoramic image according to claim 2 or 3, wherein: The obtaining of the suspension height change information of the vehicle includes: obtaining real-time suspension heights respectively detected by a plurality of suspension height detection devices of the vehicle; Acquire historical suspension heights respectively detected by the plurality of suspension height detection devices, wherein the historical suspension heights are suspension heights detected at a previous camera extrinsic parameter calibration time; determining suspension height change values ​​respectively detected by the plurality of suspension height detection devices according to the real-time suspension heights and historical suspension heights respectively detected by the plurality of suspension height detection devices; The suspension height change information is determined according to the suspension height change values ​​respectively detected by the plurality of suspension height detection devices.

5. The method for generating a panoramic image according to claim 1, wherein: The method for generating a panoramic image further includes: acquiring vehicle operation information related to a change in suspension height of the vehicle; According to the vehicle operation information, it is determined that the vehicle has a camera extrinsic parameter calibration requirement.

6. The method for generating a panoramic image according to claim 1, wherein: The vehicle detects the suspension height by a suspension height detection device, and calibrates the camera extrinsic parameters of the vehicle according to the real-time suspension height to obtain the calibrated camera extrinsic parameters, including: Obtaining the preset position coordinates of the suspension height detection device; determining the real-time position coordinates of the suspension height detection device according to the real-time suspension height and the preset position coordinates; Determining a posture transformation matrix of the vehicle according to the preset position coordinates and the real-time position coordinates; The camera extrinsic parameters of the vehicle are calibrated according to the posture transformation matrix to obtain calibrated camera extrinsic parameters.

7. The method for generating a panoramic image according to claim 6, wherein: The step of calibrating the vehicle's camera extrinsic parameters according to the pose transformation matrix to obtain calibrated camera extrinsic parameters includes: Decomposing the posture transformation matrix to obtain a rotation matrix and a translation matrix; The camera extrinsic parameters of the vehicle are calibrated according to the rotation matrix and the translation matrix to obtain calibrated camera extrinsic parameters.

8. The method for generating a panoramic image according to claim 7, wherein: The step of calibrating the camera extrinsic parameters of the vehicle according to the rotation matrix and the translation matrix to obtain the calibrated camera extrinsic parameters includes: Calibrate the rotation parameters included in the camera extrinsic parameters of the vehicle according to the rotation matrix to obtain calibrated rotation parameters; Calibrate the translation parameters included in the camera extrinsic parameters of the vehicle according to the rotation matrix and the translation parameters to obtain calibrated translation parameters; The calibrated camera extrinsics are determined according to the calibrated rotation parameters and the calibrated translation parameters.

9. The method for generating a panoramic image according to any one of claims 6 to 8, wherein: The preset position coordinates of the suspension height detection device are position coordinates calibrated during the factory production stage of the vehicle.

10. The method for generating a panoramic image according to claim 1, wherein: The calibrated camera extrinsic parameters include calibration extrinsic parameters corresponding to the multiple cameras of the vehicle, and generating a panoramic image according to the calibrated camera extrinsic parameters and images captured by the cameras, including: Capturing images of the surrounding environment of the vehicle at different viewing angles using the multiple cameras to obtain multiple images; The multiple images are stitched together according to calibration extrinsic parameters corresponding to the multiple cameras to obtain a panoramic image.

11. A device for generating a panoramic image, characterized in that: The panoramic image generation device is configured to execute the panoramic image generation method according to any one of claims 1 to 10.

12. A vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to: execute the executable instructions to implement the method for generating a panoramic image according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a panoramic image according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for generating a panoramic image according to any one of claims 1 to 10.

Images