Panoramic image determination method and device, equipment, medium and product

Abstract

The invention discloses a panoramic image determination method and device, equipment, a medium and a product. The method comprises the following steps: acquiring a current scene image, shot by at least one camera device, of an environment to which a vehicle belongs, and kinematics data and attitude data of the vehicle from a current shooting moment to a previous shooting moment; determining a scene image transformation matrix based on the kinematics data and the attitude data; acquiring a historical scene image shot by the camera device at a previous shooting moment, and performing stereo matching on the current scene image and the historical scene image based on the scene image transformation matrix and shooting parameters of the camera device to obtain a scene stereo image corresponding to the camera device; and determining and displaying a panoramic image based on the scene stereo image of each camera device. The authenticity, accuracy and reliability of panoramic image determination are improved, and the driving safety is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer processing, and in particular to a panoramic image determination method, device, equipment, medium and product. BACKGROUND

[0002] With the development of intelligent driving assistance technology, the demand of drivers for the safety and convenience of vehicles is getting higher and higher. When driving, it is usually necessary to provide the driver with panoramic images collected by the surround-view camera of the vehicle and present them on the in-vehicle screen to help the driver clearly view the situation around the vehicle, thereby improving driving safety.

[0003] At present, the traditional method for determining panoramic images usually assumes that the ground is flat, and splices multiple images on the ground plane to obtain panoramic images. However, this approach results in a large distortion in the determined panoramic images when there is a height difference or an obstacle around the vehicle, which cannot truly present the ground situation, and when the vehicle is driving on a road with steps or potholes, the driver may not be able to accurately judge the distance between the vehicle and the obstacle, thereby causing the problem of low safety. SUMMARY

[0004] The present application provides a panoramic image determination method, device, equipment, medium and product to improve the accuracy and reliability of panoramic image determination and improve driving safety.

[0005] According to an aspect of the present application, a panoramic image determination method is provided, applied to a vehicle, at least one camera device is deployed on the vehicle, and the method comprises:

[0006] During the driving of the vehicle, a current scene image of an environment to which the vehicle belongs, which is captured by at least one camera device at a current capturing time, and kinematic data and attitude data of the vehicle between the current capturing time and a previous capturing time are acquired;

[0007] Based on the kinematic data and the attitude data, a scene image transformation matrix is determined; wherein the scene image transformation matrix is used to represent the transformation matrix required when the images captured by the same camera device at the previous capturing time and at the current capturing time are overlapped;

[0008] For each camera device, a historical scene image captured by the camera device at the previous capturing time is acquired, and the current scene image and the historical scene image are stereoscopically matched based on the scene image transformation matrix and the camera parameters of the camera device to obtain a scene stereoscopic image corresponding to the camera device;

[0009] Based on the scene stereo image corresponding to each of the camera devices, a panoramic image of the environment to which the vehicle belongs is determined and displayed.

[0010] According to another aspect of the present invention, a panoramic image determining device is provided, configured in a vehicle, wherein at least one camera device is deployed on the vehicle, the device comprising:

[0011] The data acquisition module is used to acquire, during the vehicle's operation, the current scene image of the vehicle's environment captured by at least one of the camera devices at the current shooting moment, as well as the vehicle's kinematic data and attitude data between the current shooting moment and the previous shooting moment.

[0012] The transformation matrix determination module is used to determine the scene image transformation matrix based on the kinematic data and the attitude data; wherein, the scene image transformation matrix is ​​used to characterize the transformation matrix required to overlap the images captured by the same camera device at the previous shooting time and at the current shooting time;

[0013] The scene stereo image determination module is used to acquire, for each of the camera devices, a historical scene image captured by the camera device at the previous shooting time, and perform stereo matching on the current scene image and the historical scene image based on the scene image transformation matrix and the camera parameters of the camera device to obtain a scene stereo image corresponding to the camera device.

[0014] The panoramic image determination module is used to determine the scene stereoscopic image corresponding to each of the camera devices.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and a memory communicatively connected to said at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the panoramic image determination method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the panoramic image determination method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the panoramic image determination method as described in any embodiment of the present invention.

[0020] The technical solution of this invention involves acquiring, during vehicle operation, a current scene image of the vehicle's environment captured by at least one camera device at the current shooting moment, as well as kinematic and attitude data of the vehicle from the current shooting moment to the previous shooting moment; determining a scene image transformation matrix based on the kinematic and attitude data; the scene image transformation matrix characterizing the transformation matrix required to overlap images captured by the same camera device at the previous shooting moment and at the current shooting moment; acquiring historical scene images captured by the camera device at the previous shooting moment; and, based on the scene image transformation matrix and the camera parameters of the camera device, processing the current scene image... Stereo matching is performed between the current scene image and historical scene images to obtain stereo images corresponding to the camera devices. Based on the stereo images corresponding to each camera device, a panoramic image of the vehicle's environment is determined and displayed. This solves the problem of distortion and low security caused by stitching multiple images on a ground plane to determine a panoramic image in existing technologies. The new method determines the scene image transformation matrix based on the vehicle's kinematic and attitude data from the current shooting moment to the previous shooting moment. This transformation matrix accurately represents the transformation matrix required to superimpose images captured by the same camera device at the previous and current shooting moments. Furthermore, based on the scene image transformation matrix and the camera parameters of the camera devices, stereo matching is performed between the current scene image at the current shooting moment and the historical scene image at the previous shooting moment to obtain a stereo image with depth information corresponding to each camera device. Finally, based on the stereo images corresponding to each camera device, a panoramic image of the vehicle's environment is determined and displayed, improving the accuracy and reliability of panoramic image determination, ensuring the authenticity of the panoramic image, and enhancing driving safety.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a panoramic image determination method provided in Embodiment 1 of the present invention;

[0024] Figure 2This is a schematic diagram for characterizing a disparity map according to Embodiment 1 of the present invention;

[0025] Figure 3 This is a schematic diagram for characterizing a three-dimensional image of a scene according to Embodiment 1 of the present invention;

[0026] Figure 4 This is a schematic diagram for characterizing a panoramic image according to Embodiment 1 of the present invention;

[0027] Figure 5 This is a flowchart of a panoramic image determination method provided in Embodiment 2 of the present invention;

[0028] Figure 6 This is a schematic diagram of the structure of a panoramic image determination device according to Embodiment 3 of the present invention;

[0029] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the panoramic image determination method of this invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1This is a flowchart of a panoramic image determination method according to Embodiment 1 of the present invention. This embodiment is applicable to determining a panoramic image of the environment in which a vehicle belongs to assist in driving the vehicle. This method is applied to a vehicle and can be executed by a panoramic image determination device. The panoramic image determination device can be implemented in hardware and / or software and can be configured in a computing device. Figure 1 As shown, the method includes:

[0034] S110. During the vehicle's movement, acquire the current scene image of the vehicle's environment captured by at least one camera device at the current shooting moment, as well as the vehicle's kinematic data and attitude data between the current shooting moment and the previous shooting moment.

[0035] The vehicle's environment can refer to the specific scene in which the vehicle is located, such as the road scene where the vehicle is traveling, the surrounding buildings, or the parking scene where the vehicle is to be parked. For example, the environment may include at least one of the following: road, pedestrians, vehicles, buildings, signs, fences, traffic lights, trees, and parking spaces. At least one camera device may be deployed on the vehicle. For example, the number of camera devices may include multiple devices, which may be deployed at different locations on the vehicle or at different shooting angles to capture scene images of the vehicle's surrounding environment from different perspectives. For example, the camera device may be a 3D camera, fisheye camera, binocular camera, monocular camera, etc., and the specific type of camera device is not limited. Kinematic data refers to data used to characterize the vehicle's motion state; for example, kinematic data includes, but is not limited to, velocity, acceleration, and displacement. Attitude data can be data used to characterize the vehicle's attitude state in space; for example, attitude data includes, but is not limited to, the vehicle's tilt angle, pitch angle, and yaw angle.

[0036] In this embodiment, when a panoramic image of the vehicle's surrounding environment needs to be determined, image data of the vehicle's environment can be collected in real time or periodically by camera devices at different positions on the vehicle. The image collected at the current shooting moment is used as the current scene image. The current scene image can be one or multiple images. If multiple images are used, they can be captured simultaneously by camera devices at different shooting angles or positions, obtaining multi-view scene images of the environment. This avoids the problem of incomplete scene due to a single viewpoint and improves the comprehensiveness of the panoramic image determination. Simultaneously, inertial measurement units or sensors installed on the vehicle, such as wheel speed sensors, accelerometers, and gyroscopes, can be used to measure the vehicle's kinematic and attitude data in real time. All kinematic and attitude data of the vehicle from the previous shooting moment to the current shooting moment are acquired, including the vehicle's kinematic and attitude data at the previous shooting moment, the vehicle's kinematic and attitude data at the current shooting moment, and the kinematic and attitude data of the vehicle at each moment between the previous and current shooting moments.

[0037] For example, when helping vehicle A determine a panoramic image of its surrounding environment, camera devices deployed at different locations on vehicle A can capture road images from different perspectives during vehicle A's movement, serving as current scene images. These current scene images can be images from the front, rear, left, and right sides of the vehicle, respectively. A panoramic image of the environment surrounding vehicle A can be constructed using these multiple current scene images. It should be noted that the vehicle's movement can be forward acceleration or braking within the lane, or it can be reversing. The current scene images captured by each camera device at the current shooting moment are acquired. Assuming the previous shooting moment was 2 seconds ago, all kinematic and attitude data of the vehicle from the current shooting moment to 2 seconds ago are acquired.

[0038] It should be noted that after acquiring the current scene image, it can be processed, such as denoising, contrast enhancement, and edge detection, to determine the panoramic image based on the processed current scene image and improve image quality.

[0039] S120. Based on kinematic data and attitude data, determine the scene image transformation matrix.

[0040] The scene image transformation matrix is ​​used to characterize the transformation matrix required to superimpose the images captured by the same camera device at a previous shooting moment with those captured at the current shooting moment. In other words, it describes the geometric transformation relationship between two scene images (the current scene image and the historical scene image), which can include operations such as translation, rotation, and scaling. For example, it describes how the position of the same object is transformed from the historical scene image to the current scene image when the camera device moves or the scene changes.

[0041] In this embodiment, considering that the camera device is fixedly connected to the vehicle body, the movement and rotation of the vehicle from the previous shooting moment to the current shooting moment can be calculated based on kinematic data and attitude data, resulting in translation and rotation matrices. The translation and rotation matrices are then combined to obtain the scene image transformation matrix. This scene image transformation matrix is ​​then used to transform the scene image captured by the same camera device at the previous shooting moment, making it similar to the scene image captured at the current shooting moment.

[0042] In this embodiment, the kinematic data includes vehicle speed and vehicle yaw rate, and the attitude data includes vehicle body attitude offset data. In determining the scene image transformation matrix based on the kinematic data and attitude data: the rotation matrix to be used corresponding to the camera coordinate system of the camera device at the previous shooting moment can be determined based on the vehicle yaw rate in the kinematic data and the vehicle body attitude offset data in the attitude data; the translation matrix to be used corresponding to the origin of the camera coordinate system of the camera device at the previous shooting moment can be determined based on the vehicle speed and vehicle yaw rate in the kinematic data and the vehicle body attitude offset data in the attitude data; and the scene image transformation matrix is ​​determined based on the rotation matrix and the translation matrix to be used.

[0043] The vehicle yaw rate refers to the angular velocity of the vehicle's rotation about the vertical axis (z-axis) on the horizontal plane, and its unit can be radians per second (rad / s). Vehicle attitude offset data characterizes the vehicle's yaw information relative to the horizontal plane, including yaw angle, pitch angle, and roll angle to the left or right. Vehicle velocity refers to the vehicle's linear velocity on the horizontal plane, composed of velocity components along the x and y axes. The rotation matrix to be used can be used to describe the rotational movement of the camera device in space. The translation matrix to be used can be used to describe the linear movement of the camera device in space.

[0044] This can be understood as follows: based on the vehicle yaw rate from the kinematic data and the vehicle body attitude offset from the attitude data, a rotation matrix of the vehicle from the previous moment to the current moment can be calculated, which serves as the rotation matrix to be used. Based on the vehicle velocity, yaw rate, and vehicle body attitude offset data from the kinematic data, a displacement vector of the vehicle from the previous moment to the current moment can be calculated. Furthermore, the rotation matrix and translation matrix to be used are combined into a transformation matrix, which serves as the scene image transformation matrix.

[0045] For example, the scene image transformation matrix can be represented as: Where R is t n-1 Camera coordinate system at shooting time to t n The rotation matrix of the camera coordinate system at the moment of shooting, i.e., the rotation matrix to be used; T is the rotation matrix of t. n-1 Camera coordinate system at shooting time to t n The translation vector of the camera coordinate system at the moment of shooting, i.e., the translation matrix to be used. n Let t represent the nth shooting moment. n-1 This represents the (n-1)th shooting moment, which is the shooting moment preceding the nth shooting moment.

[0046] In this embodiment, the vehicle body attitude offset data includes the vehicle body roll angle, vehicle longitudinal slope, relative height between the vehicle body and the vehicle upper suspension, vehicle body roll angle, and vehicle lateral slope. During the process of determining the rotation matrix to be used corresponding to the camera coordinate system of the camera device at the previous shooting moment based on the vehicle yaw rate in the kinematic data and the vehicle body attitude offset data in the attitude data: Based on at least one of the vehicle body roll angle, vehicle longitudinal slope, and relative height between the vehicle body and the vehicle upper suspension in the vehicle body attitude offset data, a first rotation angle about the first axis of the camera coordinate system of the camera device at the previous shooting moment can be determined, and based on... A first rotation matrix is ​​determined based on a first rotation angle; a second rotation angle is determined based on at least one of the vehicle roll angle, vehicle lateral slope, and relative height between the vehicle body and the vehicle upper suspension in the vehicle body attitude offset data, and a second rotation matrix is ​​determined based on the second rotation angle; a third rotation angle is determined based on the vehicle yaw rate in the kinematic data, and a third rotation matrix is ​​determined based on the third rotation angle; and a rotation matrix to be used is determined based on the first rotation matrix, the second rotation matrix, and the third rotation matrix.

[0047] In this system, the first axis can be the Y-axis, the second axis can be the X-axis, and the third axis can be the Z-axis. The vehicle's pitch angle refers to the angle of rotation of the vehicle around its lateral axis (e.g., the Y-axis), which is the angle between the vehicle's longitudinal direction and the horizontal direction of the ground, used to characterize the vehicle's pitch state. The vehicle's longitudinal slope refers to the degree of tilt of the vehicle in its longitudinal direction (the direction of travel). It should be noted that when the vehicle is not running, the relative height between its body and the upper suspension is constant, such as 0. When the vehicle is running, with deceleration, load, acceleration, and going up or down slopes, the suspension will move up and down, and the relative height between the body and the upper suspension will change. The first rotation angle refers to the angle of rotation of the vehicle or camera device around the first axis. The first rotation matrix is ​​used to reflect the tilt changes of the vehicle or camera device in the front-to-back direction. The vehicle's roll angle can be the angle of rotation of the vehicle around its longitudinal axis (e.g., the X-axis), which is the angle between the vehicle's lateral direction and the horizontal direction of the ground, used to characterize the vehicle's roll state. Vehicle lateral slope refers to the degree of tilt of a vehicle in the lateral direction (lateral is the direction perpendicular to the vehicle's direction of travel). The second rotation angle refers to the rotation angle of the vehicle or camera device around a second axis. The second rotation matrix is ​​used to reflect changes in vehicle tilt in the left-right direction. The third rotation angle refers to the rotation angle of the vehicle or camera device around a third axis. The third rotation matrix is ​​used to reflect changes in vehicle steering.

[0048] In practical applications, at least one of the following data can be used: vehicle roll angle, vehicle longitudinal slope, or the relative height between the vehicle body and the upper suspension. This data is used to calculate the rotation matrix around the y-axis of the camera coordinate system, which serves as the first rotation matrix. Then, at least one of the following data can be used: vehicle roll angle, vehicle lateral slope, or the relative height between the vehicle body and the upper suspension. This data is used to calculate the rotation matrix around the x-axis of the camera coordinate system, which serves as the second rotation matrix. Finally, the vehicle yaw rate is used to calculate the rotation matrix around the z-axis of the camera coordinate system, which serves as the third rotation matrix. Furthermore, the first, second, and third rotation matrices are multiplied to obtain the rotation matrix to be used. This combined rotation matrix can describe the complete rotational change of the camera device from the previous moment to the current moment.

[0049] For example, the first rotation matrix can be determined based on formula (1), which can be expressed as: Among them, R y (α) represents the first rotation matrix about the y-axis of the camera coordinate system, and α represents the first rotation angle; cos represents the cosine function, and sin represents the sine function. The second rotation matrix can be determined based on formula (2), which can be expressed as: R x(γ) represents the second rotation matrix around the x-axis of the camera coordinate system, and γ represents the second rotation angle. The third rotation matrix can be determined based on formula (3), which can be expressed as: R z (β) represents the third rotation matrix about the z-axis of the camera coordinate system, and β represents the third rotation angle. The rotation matrix to be used can be determined based on formula (4); formula (4) can be expressed as R = R y (α)*R x (γ)*R z (β)(4); where R y (α) represents the revolution around t n-1 The first rotation matrix of the camera coordinate system along the y-axis, i.e., the pitch direction; R x (γ) represents the revolution around t n-1 The second rotation matrix in the roll direction, representing the x-axis of the camera coordinate system at any given time; R z (β) represents the revolution around t n-1 The third rotation matrix of the camera coordinate system at any given time is the z-axis, i.e., the yaw direction.

[0050] In this embodiment, determining a first rotation angle about a first axis of the camera coordinate system of the camera device at the previous shooting moment, based on at least one of the vehicle body tilt angle, vehicle longitudinal slope, and relative height between the vehicle body and the vehicle upper suspension in the vehicle body attitude offset data, includes: when the vehicle body attitude offset data includes the vehicle body tilt angle, determining the first rotation angle based on the vehicle body tilt angle at the current shooting moment and the vehicle body tilt angle at the previous shooting moment; when the vehicle body attitude offset data includes the vehicle longitudinal slope, determining the first rotation angle based on the vehicle longitudinal slope at the current shooting moment and the vehicle longitudinal slope at the previous shooting moment; when the vehicle body attitude offset data includes the relative height between the vehicle body and the vehicle upper suspension and the vehicle longitudinal slope, determining the first rotation angle based on the relative height and vehicle longitudinal slope at the current shooting moment, the relative height and vehicle longitudinal slope at the previous shooting moment, and the vehicle wheelbase length.

[0051] Specifically, if the vehicle attitude offset data includes the vehicle pitch angle, the pitch angle at the current shooting moment and the pitch angle at the previous shooting moment can be subtracted to obtain the rotation angle around the x-axis. If the vehicle attitude offset data includes the vehicle longitudinal slope, the longitudinal slope at the current shooting moment and the longitudinal slope at the previous shooting moment can be input into the first rotation angle determination function to obtain the rotation angle around the x-axis. If the vehicle attitude offset data includes the relative height between the vehicle body and the upper suspension and the vehicle longitudinal slope, the relative height and longitudinal slope at the current shooting moment, the relative height and longitudinal slope at the previous shooting moment, and the vehicle wheelbase length can be input into the second rotation angle determination function to obtain the rotation angle around the x-axis. The rotation angle around the x-axis is the first rotation angle.

[0052] For example, since the camera is fixed to the vehicle body, the rotation angle α of the camera around the y-axis is the rotation angle of the vehicle around the y-axis of the vehicle coordinate system, and there are at least three ways to determine the first rotation angle.

[0053] One implementation method is: if the vehicle is equipped with an IMU (Inertial Measurement Unit), the angle between the longitudinal and horizontal directions of the vehicle body (i.e., the vehicle body pitch angle) output by the IMU at two shooting moments can be used. Determine the first rotation angle α. The first rotation angle is expressed as... For the current shooting time t n The vehicle body's tilt angle is below. For the current shooting time t n-1 The vehicle body's longitudinal tilt angle.

[0054] Another implementation is as follows: If the vehicle is not equipped with an IMU, the longitudinal slope of the vehicle at two different shooting moments can be input into the first rotation angle determination function to obtain the first rotation angle. The first rotation angle determination function can be expressed as follows: t n t n-1 The longitudinal slope of the vehicle in the vehicle coordinate system at any given time; tan represents the tangent function. It should be noted that there are multiple methods for calculating the longitudinal slope of the vehicle, and this embodiment of the invention does not limit this method.

[0055] Another implementation method is: if the vehicle is equipped with a suspension height sensor, the relative height between the vehicle body and the upper suspension at the two shooting moments, as well as the vehicle's longitudinal slope, can be input into the second rotation angle determination function to obtain the first rotation angle. The second rotation angle determination function can be expressed as:

[0056]

[0057] in: For t n The relative height between the left front suspension and the vehicle body at any given time; For t n The relative height between the right front suspension and the vehicle body at any given time; For t n The relative height between the left rear suspension and the vehicle body at any given time; For t n The relative height between the right rear suspension and the vehicle body at any given time; L ab This refers to the vehicle's wheelbase length.

[0058] In this embodiment, determining a second rotation angle about the second axis of the camera coordinate system of the camera device at the previous shooting time, based on at least one of the vehicle body roll angle, vehicle lateral slope, and relative height between the vehicle body and the vehicle upper suspension in the vehicle body attitude offset data, includes: if the vehicle body attitude offset data includes the vehicle body roll angle, determining the second rotation angle based on the vehicle body roll angle at the current shooting time and the vehicle body roll angle at the previous shooting time; if the vehicle body attitude offset data includes the vehicle lateral slope, determining the second rotation angle based on the vehicle lateral slope at the current shooting time and the vehicle lateral slope at the previous shooting time; if the vehicle body attitude offset data includes the relative height between the vehicle body and the vehicle upper suspension and the vehicle lateral slope, determining the second rotation angle based on the relative height and vehicle lateral slope at the current shooting time, the relative height and vehicle lateral slope at the previous shooting time, and the vehicle wheelbase length.

[0059] Specifically, if the vehicle attitude offset data includes the vehicle roll angle, the difference between the vehicle roll angle at the current shooting moment and the vehicle roll angle at the previous shooting moment can be processed to obtain the rotation angle around the second axis. If the vehicle attitude offset data includes the vehicle lateral slope, the vehicle lateral slope at the current shooting moment and the vehicle lateral slope at the previous shooting moment can be input into the third rotation angle determination function to obtain the rotation angle around the second axis. If the vehicle attitude offset data includes the relative height between the vehicle body and the upper suspension and the vehicle lateral slope, the relative height and vehicle lateral slope at the current shooting moment, the relative height and vehicle lateral slope at the previous shooting moment, and the vehicle wheelbase length can be input into the fourth rotation angle determination function to obtain the rotation angle around the x-axis. The rotation angle around the second axis is the second rotation angle.

[0060] For example, if the vehicle is equipped with an IMU (Inertial Measurement Unit), the angle between the vehicle's lateral and horizontal directions (i.e., the vehicle's roll angle) can be obtained by the IMU outputting the angle between the vehicle's lateral and horizontal directions at two different shooting times. Determine the second rotation angle γ. The second rotation angle is expressed as... For the current shooting time t nThe body roll angle is lowered. For the current shooting time t n-1 The body roll angle.

[0061] If the vehicle is not equipped with an IMU, the lateral slope of the vehicle at the two consecutive shooting moments can be input into the third rotation angle determination function to obtain the second rotation angle. The third rotation angle determination function can be expressed as follows: t n t n-1 The lateral slope of the vehicle in the vehicle coordinate system at any given time.

[0062] If the vehicle is equipped with a suspension height sensor, the relative height between the vehicle body and the upper suspension, as well as the vehicle's lateral slope, at the two shooting moments can be input into the fourth rotation angle determination function to obtain the second rotation angle. The fourth rotation angle determination function can be expressed as:

[0063]

[0064] In this embodiment, based on the vehicle speed and yaw rate in the kinematic data and the vehicle body posture offset data in the attitude data, the translation matrix to be used corresponding to the camera coordinate system origin of the camera device at the previous shooting moment is determined, including: determining the first displacement data of the camera device coordinate system origin in the first direction at the previous shooting moment based on the vehicle speed and yaw rate in the kinematic data and the vehicle body posture offset data in the attitude data according to the first displacement determination function; determining the second displacement data of the camera device coordinate system origin in the second direction at the previous shooting moment based on the vehicle speed and yaw rate in the kinematic data and the vehicle body posture offset data in the attitude data according to the second displacement determination function; determining the third displacement data of the camera device coordinate system origin in the third direction at the previous shooting moment based on the vehicle speed and yaw rate in the kinematic data and the vehicle body posture offset data in the attitude data; and determining the translation matrix to be used based on the first displacement data, the second displacement data, and the third displacement data.

[0065] Here, the first direction can refer to the x-axis, the second direction to the y-axis, and the third direction to the z-axis. The first displacement data can refer to the displacement of the origin of the camera's coordinate system in the first direction. The second displacement data can refer to the displacement of the origin of the camera's coordinate system in the second direction. The third displacement data can refer to the displacement of the origin of the camera's coordinate system in the third direction.

[0066] In practical applications, vehicle velocity and yaw rate from kinematic data, along with vehicle body attitude offset data from attitude data, can be used as input parameters to determine the first displacement function, outputting the first displacement data of the camera's coordinate system origin in the first direction at the previous shooting moment. The vehicle velocity and yaw rate from kinematic data, along with the vehicle body attitude offset data from attitude data, are input into the second displacement function, outputting the second displacement data of the camera's coordinate system origin in the second direction at the previous shooting moment. Based on the vehicle body attitude offset data from attitude data, the vehicle velocity in the vertical direction (z-direction) from the kinematic data is integrated to obtain the third displacement data of the camera's coordinate system origin in the third direction at the previous shooting moment. The first, second, and third displacement data are combined to obtain the translation matrix to be used.

[0067] For example, the second displacement determination function can be expressed as,

[0068] Among them, T y This represents the second displacement data, u represents the vehicle speed, and ω represents the displacement data. yaw Indicates the vehicle's yaw rate; This can be expressed as the change in the vehicle's yaw angle, which is represented using t. n to t n-1 Integrate the vehicle's yaw rate between the two points to obtain the change in the vehicle's yaw angle; sin -1 Represents the arcsine function; i x Indicates t n to t n-1 The longitudinal slope of the vehicle at each time point between these points. The third displacement data can be determined based on formula (5), which can be expressed as: T z The third displacement data is obtained by integrating the vehicle's velocity in the vertical (z-direction) based on the vehicle's longitudinal slope. The translation matrix T to be used can be determined based on formula (6), which can be expressed as: Wherein: T x This is represented as the first displacement data of the origin of the coordinate axis in the first direction; T y This represents the second displacement data of the coordinate axis origin in the second direction; T z This represents the third displacement data of the origin of the coordinate axis in the third direction.

[0069] In this embodiment, based on the first displacement determination function, and using the vehicle speed and yaw rate in the kinematic data and the vehicle body attitude offset data in the attitude data, the first displacement data of the origin of the camera device's coordinate system in the first direction at the previous shooting moment is determined. This includes: inputting the kinematic data and attitude data into the first displacement determination function to obtain the change in vehicle yaw angle; and determining the first displacement data in the first direction based on the change in vehicle yaw angle, the vehicle speed in the kinematic data, and the vehicle longitudinal slope in the vehicle body attitude offset data.

[0070] The change in vehicle yaw angle is determined based on the vehicle yaw rate from the kinematic data.

[0071] Specifically, the function for determining the first displacement can be expressed as follows: Among them, T x This represents the first displacement data, u represents the vehicle speed, and ω represents the first displacement data. yaw Indicates the vehicle's yaw rate; This can be expressed as the change in the vehicle's yaw angle, which is represented using t. n to t n-1 Integrate the vehicle's yaw rate between the two points to obtain the change in the vehicle's yaw angle; sin -1 Represents the arcsine function; i x Indicates t n to t n-1 The longitudinal slope of the vehicle at each time point between these points.

[0072] S130. For each camera device, acquire the historical scene image captured by the camera device at the previous shooting moment, and perform stereo matching on the current scene image and the historical scene image based on the scene image transformation matrix and the camera parameters of the camera device to obtain the scene stereo image corresponding to the camera device.

[0073] Camera parameters refer to parameters related to the characteristics of the camera device itself, which can be obtained through calibration at the factory. For example, camera parameters may include, but are not limited to, focal length, optical center position, pixel size, radial distortion coefficient, tangential distortion coefficient, etc. A scene stereo image refers to an image obtained after stereo matching processing. In a scene stereo image, each point not only has color information but also an added depth value, which represents the distance of the scene object represented by that point from the camera device. It should be noted that the method for determining the scene stereo image corresponding to each camera device is the same. The following explanation uses the determination of the scene stereo image corresponding to any one of the camera devices as an example.

[0074] In this embodiment, for a given camera device, it can acquire historical scene images of the scene to which the vehicle belongs, captured at a previous shooting moment. For example, if the camera device captures images at a frequency of 30 frames per second, then the image captured at the current moment is the current scene image, while the image captured in the previous frame (approximately 1 / 30 of a second ago) or the image captured in the previous second is the historical scene image. It records the scene information within the field of view of the camera device at the previous moment. Furthermore, during stereo matching, camera parameters can be used to process the image coordinates, such as correction and transformation. For example, camera parameters can be used to convert the coordinates of two scene images into normalized image plane coordinates, eliminating the influence of factors such as focal length and optical center position on the image coordinates. The positions of each pixel in the historical scene image can be transformed based on the scene image transformation matrix. The transformed pixel positions in the historical scene image are compared with the pixel positions in the current scene image to determine the pixel representing the same object in the same scene in both images. For example, an object appears at position A in a historical scene image, but due to the movement of the camera or the object itself, it appears at position B in the current scene image. A scene image transformation matrix can be used to transform this position in the historical scene image; the transformed position is position B in the current scene image. Furthermore, the distance of a pixel to the camera can be determined based on the positional difference of pixels at the same location in the two images. A depth map is then determined based on the distance to each pixel. Fusing the depth map with the color information of the current scene image yields a stereoscopic image of the scene. For example, the depth map can be used as a channel of the image (e.g., a depth channel). Combining the RGB color channels of the current scene image with the depth channel creates a stereoscopic image of the scene, allowing both color and depth information to be observed.

[0075] For example, see Figure 2 The vehicle from t n-1 The vehicle's position at time t (i.e., the previous shooting time) has reached t n The vehicle's position at time t (i.e., the current shooting time). Area S in the diagram represents the portion covered by the combined perspective of the surround-view cameras from two points prior to this time. n Time perspective and t n-1 A virtual binocular stereoscopic vision is formed by considering the perspectives at different times. The scene images acquired by the camera devices at two different times can undergo the same noise reduction and distortion correction processing based on the camera parameters. Then, stereo matching calculations are performed on the images at the two times based on the scene image transformation matrix to obtain a disparity map between the two images. The disparity map can characterize the two scene images S. f The pixel difference between offset regions of the same pixel within the same area. Furthermore, the disparity map can be used as a depth map to determine the stereoscopic image of the scene based on the depth map and the color information of the pixels.

[0076] To improve the accuracy and efficiency of scene stereo image determination, a machine learning model can be trained in advance based on scene images and corresponding stereo images captured by the camera device at adjacent times. After acquiring the historical scene images captured by the camera device at the previous shooting time and the current scene image at the current shooting time, the historical scene images, the current scene images, the scene image transformation matrix, and the camera parameters of the camera device are input into the machine learning model to obtain the scene stereo image corresponding to the camera device.

[0077] For example, based on the camera parameters of the camera device and the scene image transformation matrix, t can be... n With t n-1 The scene images acquired in real time are subjected to stereo matching to obtain a stereo image of the scene from the perspective of the camera device, as illustrated in the figure. Correspondingly, stereo images of the scene from the perspectives of multiple camera devices can be obtained.

[0078] S140. Based on the stereoscopic image of the scene corresponding to each camera device, determine and display the panoramic image of the environment in which the vehicle belongs.

[0079] Panoramic images refer to images that cover a wider area (such as 360 degrees) by stitching together three-dimensional images of a scene from multiple perspectives.

[0080] In this embodiment, color differences exist between the scene stereo images due to variations in lighting conditions or differing color response characteristics between different camera devices. To ensure color consistency in the stitched panoramic image, a color correction algorithm can be employed. For example, by calculating the color transformation matrix between different images, the colors of all scene stereo images can be adjusted to a unified color space, resulting in a processed scene stereo image. Furthermore, based on the deployment position of each camera device on the vehicle, each processed scene stereo image can be stitched together to obtain a panoramic image of the vehicle's environment. Alternatively, feature extraction can be performed on the scene stereo images to determine the relative positional relationships between adjacent images. Then, based on these relative positional relationships, each scene stereo image can be stitched together to obtain a panoramic image. Further, the panoramic image can be displayed on a target terminal. For example, the target terminal could be the vehicle's central control screen, the display terminal of a monitoring system, or a mobile phone terminal. The driver can then visually observe the environment around the vehicle from all directions using the panoramic image.

[0081] For example, see Figure 3 , Figure 3This can be represented as a stereoscopic image of the scene from a single shooting perspective. The stereoscopic images acquired from four perspectives can cover most of the area around the vehicle. Real-time calculations can be performed on images from adjacent time points, and the stereoscopic images can be iterated and filtered to obtain a stable panoramic image of the vehicle's perimeter with color information. A schematic diagram of the panoramic image can be found in [reference needed]. Figure 4 .

[0082] In this embodiment, during the process of determining and displaying the panoramic image of the vehicle's environment based on the scene stereo image corresponding to each camera device, surface fitting can be performed on the scene stereo image corresponding to each camera device to obtain the panoramic image of the vehicle's environment; the panoramic image is then projected and displayed in the top-down direction of the vehicle's coordinate system.

[0083] The vehicle coordinate system refers to the coordinate system associated with the vehicle itself. For example, the vehicle coordinate system can have its origin at a fixed point on the vehicle (such as the vehicle's center of mass), the vehicle's direction of travel as the positive X-axis, the vehicle's lateral direction as the Y-axis, and the direction perpendicular to the vehicle's plane as the Z-axis. The top-view direction of the vehicle coordinate system refers to the direction looking down from directly above the vehicle. This direction is used to display a panoramic image of the vehicle's surrounding environment. The panoramic image includes the three-dimensional shape of the environment surrounding the vehicle. For example, the panoramic image can be represented as a smooth three-dimensional surface, depicting the curvature of roads, the outlines of buildings, and other complex three-dimensional structures.

[0084] Specifically, for the stereoscopic image of the scene corresponding to each camera device, the points, depth information, and color information can be extracted. These data points are used as input data for a surface fitting method. Based on this method, a surface fitting is performed using the input data to obtain a panoramic image of the vehicle's environment. Optionally, surface fitting methods include, but are not limited to, polynomial fitting, spline fitting, and point cloud-based surface reconstruction methods. It should be noted that during surface fitting, overlapping image areas from adjacent camera devices can be smoothly transitioned using interpolation or fusion algorithms to avoid obvious stitching artifacts. Furthermore, the fitted panoramic image of the vehicle's environment can be projected onto a top-down 3D model in the vehicle coordinate system to obtain a top-down projection of the panoramic image. During projection, height information can be encoded, for example, using different colors or grayscale values ​​to represent different heights, so as to display the 3D information of the environment in the panoramic image. Alternatively, a pixelation method can be used to map each point in the panoramic image to a pixel location in the image and assign a corresponding color or grayscale value based on the point's height information. For example, lower points (such as the ground) can be represented by dark colors, while higher points (such as the top of a building) can be represented by light colors, allowing the panoramic image to intuitively display the layout and height information of the vehicle's surroundings. Furthermore, the generated panoramic image can be displayed on the vehicle's display devices, such as the central control screen, driver assistance system display screen, or monitoring system display terminal. During the display, auxiliary markers can be added to the panoramic image, such as vehicle outlines, lane lines, icons of other vehicles and pedestrians, and directional arrows, to help drivers better understand the vehicle's location and the spatial layout of its surroundings.

[0085] For example, surface fitting is performed on the 3D image of the scene to obtain the colored surface information around the vehicle, i.e., the panoramic image. The panoramic image is then projected onto the top-view direction in the vehicle coordinate system to obtain a top-down view based on the real 3D scene reconstruction, thus solving the distortion problem caused by the reliance on the horizon hypothesis in traditional panoramic imaging functions. The panoramic image can be viewed from multiple perspectives on the vehicle, and users can drag the viewpoint on the terminal screen to view the panoramic image.

[0086] The technical solution provided by this invention acquires, during vehicle operation, a current scene image of the vehicle's environment captured by at least one camera device at the current shooting moment, as well as kinematic and attitude data of the vehicle from the current shooting moment to the previous shooting moment; based on the kinematic and attitude data, a scene image transformation matrix is ​​determined; the scene image transformation matrix is ​​used to characterize the transformation matrix required to overlap the images captured by the same camera device at the previous shooting moment and at the current shooting moment; historical scene images captured by the camera device at the previous shooting moment are acquired; and based on the scene image transformation matrix and the camera parameters of the camera device, the current scene image is transformed. Stereo matching is performed between the current scene image and historical scene images to obtain stereo images corresponding to the camera devices. Based on the stereo images corresponding to each camera device, a panoramic image of the vehicle's environment is determined and displayed. This solves the problem of distortion and low security caused by stitching multiple images on a ground plane to determine a panoramic image in existing technologies. The new method determines the scene image transformation matrix based on the vehicle's kinematic and attitude data from the current shooting moment to the previous shooting moment. This transformation matrix accurately represents the transformation matrix required to superimpose images captured by the same camera device at the previous and current shooting moments. Furthermore, based on the scene image transformation matrix and the camera parameters of the camera devices, stereo matching is performed between the current scene image at the current shooting moment and the historical scene image at the previous shooting moment to obtain a stereo image with depth information corresponding to each camera device. Finally, based on the stereo images corresponding to each camera device, a panoramic image of the vehicle's environment is determined and displayed, improving the accuracy and reliability of panoramic image determination, ensuring the authenticity of the panoramic image, and enhancing driving safety.

[0087] Example 2

[0088] As an optional embodiment of the above embodiments, specific application scenario examples are provided to enable those skilled in the art to further understand the technical solutions of the embodiments of the present invention. Specifically, please refer to the following detailed content.

[0089] In this embodiment, see Figure 5This can be achieved by capturing scene images at least at two shooting moments using multiple surround-view cameras deployed on the vehicle while it is moving. Within each image channel, a binocular image is composed of these two scene images (i.e., the current scene image and a historical scene image). Vehicle kinematic information (i.e., kinematic data and attitude data) is acquired, and the scene image transformation matrix of the camera devices is calculated. Furthermore, based on the scene image transformation matrix and the camera parameters of the camera devices, stereo matching is performed on the current scene image and the historical scene image to obtain a stereo image of the scene corresponding to the camera devices, achieving virtual binocular stereo vision perception, i.e., acquiring a stereo image within a single viewpoint, that is, a road surface elevation curve image within the single viewpoint's field of view. Correspondingly, scene stereo images are acquired from at least four shooting viewpoint directions, i.e., acquiring color point cloud information from multiple viewpoints around the vehicle. Furthermore, the system stitches, filters, and performs surface fitting on stereoscopic images of the scene from multiple perspectives within the vehicle coordinate system to obtain a panoramic image of the vertical (top-down) panoramic view within the vehicle coordinate system. This image is used for displaying the panoramic view from inside the vehicle, restoring objects in different perspectives to their true positions and eliminating distortion caused by horizon calibration. In addition, the stereoscopic information in the panoramic image can be overlaid onto the panoramic image interface using contour lines or elevation heatmaps, based on the vehicle's ground contact point height. This eliminates image distortion caused by elevation issues without increasing hardware costs, and overlays road elevation information onto the display interface, avoiding safety hazards caused by potholes, shoulders, narrow scenes, etc., thus improving driving safety.

[0090] The technical solution of this embodiment acquires image information at least at two moments while the vehicle is moving, based on images from a vehicle panoramic imaging camera (surround-view camera). Within each image channel, a binocular image is formed based on the image information from the two moments. Vehicle kinematic information is acquired to calculate the extrinsic parameters of the virtual binocular camera, achieving virtual binocular stereoscopic vision perception. This means acquiring stereoscopic information within a single viewpoint, i.e., the road surface elevation curve within the single viewpoint's field of view. It also acquires road surface stereoscopic information from at least four viewpoints, and then stitches the road surface elevation curves from multiple viewpoints together in the vehicle coordinate system to obtain the road surface elevation curve of the vertical panoramic view in the vehicle coordinate system. This is used for displaying the panoramic image view from inside the vehicle, restoring objects within the viewpoint to their true positions and eliminating distortion caused by horizon calibration. Furthermore, the stereoscopic information is overlaid on the panoramic image interface using contour lines or elevation heatmaps, with the vehicle's ground contact point height as a reference. To achieve this without increasing hardware costs, software algorithms are used to eliminate image distortion caused by elevation in traditional panoramic images. Road elevation information is then overlaid on the display interface to avoid safety hazards caused by potholes, shoulders, narrow scenes, etc., thereby improving driving safety. (Example 3)

[0091] Figure 6This is a schematic diagram of the structure of a panoramic image determination device according to Embodiment 3 of the present invention. Figure 6 As shown, the device is configured in a vehicle, on which at least one camera device is deployed. The device includes: a data acquisition module 210, a transformation matrix determination module 220, a scene stereo image determination module 230, and a panoramic image determination module 240.

[0092] The data acquisition module 210 is used to acquire, during vehicle operation, the current scene image of the vehicle's environment captured by at least one of the cameras at the current shooting moment, as well as the kinematic data and attitude data of the vehicle between the current shooting moment and the previous shooting moment; the transformation matrix determination module 220 is used to determine a scene image transformation matrix based on the kinematic data and attitude data; wherein, the scene image transformation matrix is ​​used to characterize the transformation matrix required to overlap the images captured by the same camera at the previous shooting moment and the current shooting moment; the scene stereo image determination module 230 is used to acquire, for each camera, a historical scene image captured by the camera at the previous shooting moment, and perform stereo matching on the current scene image and the historical scene image based on the scene image transformation matrix and the camera parameters of the camera to obtain a scene stereo image corresponding to the camera; the panoramic image determination module 240 is used to determine and display a panoramic image of the vehicle's environment based on the scene stereo image corresponding to each camera.

[0093] The technical solution of this embodiment involves acquiring, during vehicle operation, a current scene image of the vehicle's environment captured by at least one camera device at the current shooting moment, as well as kinematic and attitude data of the vehicle from the current shooting moment to the previous shooting moment; determining a scene image transformation matrix based on the kinematic and attitude data; the scene image transformation matrix characterizing the transformation matrix required to overlap the images captured by the same camera device at the previous shooting moment with those captured at the current shooting moment; acquiring historical scene images captured by the camera device at the previous shooting moment; and, based on the scene image transformation matrix and the camera parameters of the camera device, processing the current scene image and... Historical scene images are stereo-matched to obtain stereo images of the scene corresponding to each camera device. Based on the stereo images of the scene corresponding to each camera device, a panoramic image of the vehicle's environment is determined and displayed. This solves the problem of distortion and low security caused by stitching multiple images on a ground plane to determine a panoramic image in existing technologies. It achieves this by determining the scene image transformation matrix based on the vehicle's kinematic and attitude data from the current shooting moment to the previous shooting moment. This scene image transformation matrix can accurately represent the transformation matrix required to superimpose the images captured by the same camera device at the previous shooting moment and the current shooting moment. Furthermore, based on the scene image transformation matrix and the camera parameters of the camera devices, stereo matching is performed between the current scene image at the current shooting moment and the historical scene image at the previous shooting moment to obtain a stereo image of the scene with depth information corresponding to each camera device. Based on the stereo images of the scene corresponding to each camera device, a panoramic image of the vehicle's environment is determined and displayed, improving the accuracy and reliability of panoramic image determination, ensuring the authenticity of the panoramic image, and enhancing driving safety.

[0094] Optionally, based on the above-described apparatus, the transformation matrix determination module 220 includes:

[0095] The rotation matrix determination unit is used to determine the rotation matrix to be used corresponding to the camera coordinate system of the camera device at the previous shooting time, based on the vehicle yaw rate in the kinematic data and the vehicle body attitude offset data in the attitude data.

[0096] The translation matrix determination unit is used to determine the translation matrix to be used corresponding to the origin of the camera coordinate system of the camera device at the previous shooting time, based on the vehicle speed and vehicle yaw rate in the kinematic data and the vehicle body attitude offset data in the attitude data.

[0097] The scene image transformation matrix determination unit is used to determine the scene image transformation matrix based on the rotation matrix to be used and the translation matrix to be used.

[0098] Based on the above-described apparatus, optionally, the rotation matrix determination unit to be used includes:

[0099] The first rotation matrix determination unit is used to determine a first rotation angle about the first axis of the camera coordinate system of the camera device at the previous shooting time based on at least one of the vehicle body tilt angle, vehicle longitudinal slope and relative height between the vehicle body and the vehicle upper suspension in the vehicle body attitude offset data, and to determine the first rotation matrix based on the first rotation angle.

[0100] The second rotation matrix determination unit is used to determine a second rotation angle about the second axis of the camera coordinate system of the camera device at the previous shooting time based on at least one of the vehicle body roll angle, vehicle lateral slope and relative height between the vehicle body and the vehicle upper suspension in the vehicle body attitude offset data, and to determine a second rotation matrix based on the second rotation angle.

[0101] The third rotation matrix determination unit is used to determine the third rotation angle of the third axis of the camera coordinate system of the camera device around the previous shooting time based on the vehicle yaw rate in the kinematic data, and to determine the third rotation matrix based on the third rotation angle.

[0102] The rotation matrix to be used determines the sub-unit, which is used to determine the rotation matrix to be used based on the first rotation matrix, the second rotation matrix and the third rotation matrix.

[0103] Based on the above-described apparatus, optionally, the first rotation matrix determining unit includes:

[0104] The first rotation angle determination unit is used to determine the first rotation angle based on the vehicle tilt angle at the current shooting time and the vehicle tilt angle at the previous shooting time, when the vehicle attitude offset data includes the vehicle pitch angle.

[0105] The second rotation angle determination unit is used to determine the first rotation angle based on the vehicle longitudinal slope at the current shooting time and the vehicle longitudinal slope at the previous shooting time, when the vehicle body posture offset data includes the vehicle longitudinal slope.

[0106] The third rotation angle determination unit is used to determine the first rotation angle based on the relative height between the vehicle body and the upper suspension and the longitudinal slope of the vehicle, as well as the relative height and longitudinal slope of the vehicle at the current shooting time and the relative height and longitudinal slope of the vehicle at the previous shooting time and the vehicle wheelbase length, when the vehicle body posture offset data includes the relative height between the vehicle body and the upper suspension and the longitudinal slope of the vehicle.

[0107] Based on the above-described apparatus, the optional translation matrix determination unit to be used includes:

[0108] The first displacement data determination unit is used to determine the first displacement data of the origin of the coordinate system of the camera device in the first direction at the previous shooting time based on the first displacement determination function, the vehicle speed and vehicle yaw rate in the kinematic data and the body posture offset data in the posture data;

[0109] The second displacement data determination unit is used to determine the second displacement data of the origin of the coordinate system of the camera device in the second direction at the previous shooting moment, based on the vehicle speed and vehicle yaw rate in the kinematic data and the body posture offset data in the posture data, according to the second displacement determination function;

[0110] The third displacement data determination unit is used to determine the third displacement data of the origin of the camera device in the third direction at the previous shooting moment based on the vehicle speed in the kinematic data and the vehicle longitudinal slope in the attitude data.

[0111] The translation matrix to be used determines the sub-unit, which is used to determine the translation matrix to be used based on the first displacement data, the second displacement data, and the third displacement data.

[0112] Based on the above-mentioned device, optionally, the first displacement data determination unit includes:

[0113] The change determination unit is used to input the kinematic data and the attitude data into the first displacement determination function to obtain the change in vehicle yaw angle; wherein the change in vehicle yaw angle is determined based on the vehicle yaw rate in the kinematic data;

[0114] The first displacement data determination subunit is used to determine the first displacement data in the first direction based on the change in the vehicle yaw angle, the vehicle speed in the kinematic data, and the vehicle longitudinal slope in the vehicle body attitude offset data.

[0115] Based on the above-described device, optionally, the panoramic image determination module 240 includes:

[0116] A panoramic image determination unit is used to perform surface fitting on the scene stereo image corresponding to each of the camera devices to obtain a panoramic image of the environment to which the vehicle belongs;

[0117] A panoramic image display unit is used to project and display a panoramic image from a top-down perspective in the vehicle's coordinate system.

[0118] The panoramic image determination device provided in the embodiments of the present invention can execute the panoramic image determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0119] Example 4

[0120] Figure 7 This is a schematic diagram of the structure of an electronic device implementing the panoramic image determination method of this invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0121] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory 12 or a random access memory 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 12 or loaded from storage unit 18 into the random access memory 13. The random access memory 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, read-only memory 12, and random access memory 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0122] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0123] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as panoramic image determination methods.

[0124] In some embodiments, the panoramic image determination method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via read-only memory 12 and / or communication unit 19. When the computer program is loaded into random access memory 13 and executed by processor 11, one or more steps of the panoramic image determination method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the panoramic image determination method by any other suitable means (e.g., by means of firmware).

[0125] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0129] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0130] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0131] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from read-only memory 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0132] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the panoramic image determination method provided in any embodiment of this invention.

[0133] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0135] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining panoramic images, characterized in that, Applied to a vehicle, wherein at least one camera device is deployed on the vehicle, the method includes: During the vehicle's movement, the system acquires a current scene image of the vehicle's environment captured by at least one of the cameras at the current shooting moment, as well as kinematic and attitude data of the vehicle between the current shooting moment and the previous shooting moment. Based on the kinematic data and attitude data, a scene image transformation matrix is ​​determined; wherein, the scene image transformation matrix is ​​used to characterize the transformation matrix required to overlap the images captured by the same camera device at the previous shooting time and at the current shooting time; For each of the camera devices, a historical scene image captured by the camera device at the previous shooting time is acquired. Based on the scene image transformation matrix and the camera parameters of the camera device, a stereo matching is performed on the current scene image and the historical scene image to obtain a stereo scene image corresponding to the camera device. Based on the scene stereo image corresponding to each of the camera devices, a panoramic image of the environment to which the vehicle belongs is determined and displayed.

2. The method according to claim 1, characterized in that, The step of determining the scene image transformation matrix based on the kinematic data and pose data includes: Based on the vehicle yaw rate in the kinematic data and the vehicle body attitude offset data in the attitude data, the rotation matrix to be used corresponding to the camera coordinate system of the camera device at the previous shooting time is determined. Based on the vehicle speed and yaw rate in the kinematic data and the vehicle body posture offset data in the posture data, the translation matrix to be used corresponding to the origin of the camera coordinate system of the camera device at the previous shooting time is determined. The scene image transformation matrix is ​​determined based on the rotation matrix to be used and the translation matrix to be used.

3. The method according to claim 2, characterized in that, The step of determining the rotation matrix to be used corresponding to the camera coordinate system of the camera device at the previous shooting moment based on the vehicle yaw rate in the kinematic data and the vehicle body attitude offset data in the attitude data includes: Based on at least one of the vehicle body tilt angle, vehicle longitudinal slope, and relative height between the vehicle body and the vehicle upper suspension in the vehicle body attitude offset data, a first rotation angle about the first axis of the camera coordinate system of the camera device about the previous shooting time is determined, and a first rotation matrix is ​​determined based on the first rotation angle. Based on at least one of the vehicle body roll angle, vehicle lateral slope, and relative height between the vehicle body and the upper suspension in the vehicle body attitude offset data, a second rotation angle about the second axis of the camera coordinate system of the camera device about the previous shooting time is determined, and a second rotation matrix is ​​determined based on the second rotation angle. Based on the vehicle yaw rate in the kinematic data, a third rotation angle is determined about the third axis of the camera coordinate system of the camera device around the previous shooting time, and a third rotation matrix is ​​determined based on the third rotation angle. Based on the first rotation matrix, the second rotation matrix, and the third rotation matrix, the rotation matrix to be used is determined.

4. The method according to claim 3, characterized in that, The determination of a first rotation angle about a first axis of the camera coordinate system of the imaging device at the previous shooting time, based on at least one of the vehicle body tilt angle, vehicle longitudinal slope, and relative height between the vehicle body and the vehicle upper suspension in the vehicle body attitude offset data, includes: If the vehicle body attitude offset data includes the vehicle body pitch angle, the first rotation angle is determined based on the vehicle body pitch angle at the current shooting time and the vehicle body pitch angle at the previous shooting time. If the vehicle body posture offset data includes the vehicle longitudinal slope, the first rotation angle is determined based on the vehicle longitudinal slope at the current shooting time and the vehicle longitudinal slope at the previous shooting time. When the vehicle body attitude offset data includes the relative height between the vehicle body and the vehicle upper suspension and the vehicle longitudinal slope, the first rotation angle is determined based on the relative height and the vehicle longitudinal slope at the current shooting time, the relative height and the vehicle longitudinal slope at the previous shooting time, and the vehicle wheelbase length.

5. The method according to claim 2, characterized in that, The step of determining the translation matrix to be used corresponding to the origin of the camera coordinate system of the camera device at the previous shooting time, based on the vehicle speed and yaw rate in the kinematic data and the vehicle body attitude offset data in the attitude data, includes: Based on the first displacement determination function, and based on the vehicle speed and vehicle yaw rate in the kinematic data and the vehicle body posture offset data in the posture data, the first displacement data of the origin of the coordinate system of the camera device in the first direction at the previous shooting time is determined. Based on the second displacement determination function, and based on the vehicle speed and vehicle yaw rate in the kinematic data and the vehicle body posture offset data in the posture data, the second displacement data of the origin of the coordinate system of the camera device in the second direction at the previous shooting time is determined; Based on the vehicle speed in the kinematic data and the vehicle longitudinal slope in the attitude data, the third displacement data of the origin of the camera device in the third direction at the previous shooting time is determined. Based on the first displacement data, the second displacement data, and the third displacement data, the translation matrix to be used is determined.

6. The method according to claim 5, characterized in that, The determination of the first displacement data of the origin of the camera device's coordinate system in the first direction at the previous shooting moment, based on the vehicle velocity and yaw rate in the kinematic data and the vehicle body posture offset data in the posture data, according to the first displacement determination function, includes: The kinematic data and the attitude data are input into a first displacement determination function to obtain the change in vehicle yaw angle; wherein the change in vehicle yaw angle is determined based on the vehicle yaw rate in the kinematic data; Based on the change in vehicle yaw angle, the vehicle speed in the kinematic data, and the vehicle longitudinal slope in the vehicle body attitude offset data, the first displacement data in the first direction is determined.

7. The method according to claim 1, characterized in that, The step of determining and displaying a panoramic image of the environment in which the vehicle belongs based on the scene stereoscopic image corresponding to each of the camera devices includes: By performing surface fitting on the scene stereo image corresponding to each of the camera devices, a panoramic image of the environment to which the vehicle belongs is obtained; The panoramic image is projected and displayed from the top-down view of the vehicle's coordinate system.

8. A panoramic image determining device, characterized in that, Configured in a vehicle, wherein at least one camera device is deployed on the vehicle, the device comprising: The data acquisition module is used to acquire, during the vehicle's operation, the current scene image of the vehicle's environment captured by at least one of the camera devices at the current shooting moment, as well as the vehicle's kinematic data and attitude data between the current shooting moment and the previous shooting moment. The transformation matrix determination module is used to determine the scene image transformation matrix based on the kinematic data and the attitude data; wherein, the scene image transformation matrix is ​​used to characterize the transformation matrix required to overlap the images captured by the same camera device at the previous shooting time and at the current shooting time; The scene stereo image determination module is used to acquire, for each of the camera devices, a historical scene image captured by the camera device at the previous shooting time, and perform stereo matching on the current scene image and the historical scene image based on the scene image transformation matrix and the camera parameters of the camera device to obtain a scene stereo image corresponding to the camera device. The panoramic image determination module is used to determine the scene stereoscopic image based on the scene stereoscopic image corresponding to each of the camera devices.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which is executed by the at least one processor to enable the at least one processor to perform the panoramic image determination method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the panoramic image determination method according to any one of claims 1-7.

Images