Method, device, computer system and storage medium for generating target panoramic image

Abstract

The present application provides a method, device, computer system and storage medium for generating a target panoramic image, the method comprising obtaining an initial panoramic image at the current moment and preprocessing the initial panoramic image to obtain a first intermediate image; extracting the movement amount of a preset feature in the initial panoramic image, and calculating the actual movement amount of the target vehicle according to the movement amount; obtaining the state data of the target vehicle at the current moment, and correcting the state data according to the actual movement amount; matching the first intermediate image with the vehicle chassis image according to the corrected state data to obtain a second intermediate image; fusing the second intermediate image to obtain a target panoramic image containing the vehicle chassis image. The present application scheme can eliminate the problem of stitching image distortion caused by the mismatch between wheel speed and actual vehicle speed, thereby improving the accuracy and reliability of the stitching of the vehicle transparent chassis image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, computer system and storage medium for generating a target panoramic image. Background Art

[0002] The accurate estimation of the vehicle's own position has a decisive influence on the stitching effect of the transparent bottom of the vehicle. If the error of the vehicle's own position in time or position is large, it will cause the panoramic image and the bottom image to be misaligned when stitching, and will also cause the bottom image itself to be distorted. The measurement of the vehicle's own position is difficult to obtain directly through the vehicle itself. As an alternative, it is possible to consider inferring the position of the vehicle body through the direction and speed of the vehicle, but it is very expensive to directly install a sensor to measure the vehicle speed on the vehicle. Therefore, it is usually used to measure the speed of the wheel speedometer to infer the vehicle speed, thereby indirectly inferring and calculating the vehicle's own position.

[0003] In the past, the application of wheel speed prediction of vehicle position was mainly aimed at auxiliary driving functions such as obstacle avoidance and reversing images. The results of wheel prediction mostly issued an alarm or told the driver the predicted position of the vehicle through images. Because this type of function is only a prompt or alarm, it ultimately depends on the driver to judge or correct it. Therefore, this type of work has a high tolerance for errors and can ignore the deviation caused by the wheel angle signal error. However, the output of transparent chassis stitching is the stitching result of the actual bottom image and the panoramic image. Therefore, this method has very high requirements on the accuracy of the estimated value of the vehicle position. The quality of the stitching effect is very intuitively presented to the driver, and the driver cannot correct the result of the stitching image through driving actions. In addition, the deviation of the transparent chassis will become more obvious due to the accumulation over time, making the generated image lose the function of prompting the driver. Therefore, it is particularly important to calculate the precise position of the vehicle through the vehicle speed signal.

[0004] Furthermore, in practical applications, the output of the wheel speed meter itself has been through prediction, filtering and other algorithms, resulting in different calculation accuracy of the vehicle speed at high and low speeds. In addition, when turning or accelerating or decelerating, the comprehensive analysis of the results of multiple wheel speed meters such as the left and right wheels will also amplify the calculation errors introduced by the aforementioned algorithms, resulting in the stitched image of the vehicle chassis being normal when driving, but deteriorating when turning. It is normal when driving at high speeds, but unrecognizable at low speeds. Therefore, a calculation method is needed that can unify the vehicle wheel speed meter in different situations such as left and right wheels, high and low speeds, left and right turns, etc., so as to bring high-precision vehicle position prediction results and ensure that the stitched image will not be distorted. Summary of the invention

[0005] In order to solve the problems of the prior art, the present invention provides a method, device, computer system and storage medium for generating a target panoramic image, so as to solve the problems existing in the prior art that the vehicle's own position is difficult to measure directly by the vehicle itself, the wheel speed sensor and the actual vehicle speed do not match each other resulting in distortion of the spliced ​​image, and the spliced ​​bottom pattern cannot provide valuable prompt information to the driver.

[0006] In order to solve one or more of the above technical problems, the technical solution adopted in this application is:

[0007] In a first aspect, a method for generating a target panoramic image is provided, the method comprising:

[0008] S100: Acquire an initial panoramic image at a current moment and preprocess the initial panoramic image to obtain a first intermediate image;

[0009] S200: extracting the movement amount of the preset feature in the initial panoramic image, and calculating the actual movement amount of the target vehicle according to the movement amount;

[0010] S300: Acquire the state data of the target vehicle at the current moment, and correct the state data according to the actual movement amount;

[0011] S400: performing vehicle chassis image matching on the first intermediate image according to the corrected state data to obtain a second intermediate image;

[0012] S500: Fusing the second intermediate images obtained by matching at different positions to obtain a target panoramic image including a vehicle chassis image.

[0013] In a specific embodiment, extracting the movement amount of the preset feature in the initial panoramic image and calculating the actual movement amount of the target vehicle according to the movement amount includes:

[0014] Calculating the movement amount of the preset features in the initial panoramic image based on a dense optical flow algorithm;

[0015] Obtaining a rotation center of the target vehicle according to the movement amount;

[0016] The target vehicle is subjected to motion decomposition according to the rotation center and an actual motion amount of the target vehicle is calculated.

[0017] In a specific embodiment, extracting the movement amount of the preset feature in the initial panoramic image and calculating the actual movement amount of the target vehicle according to the movement amount includes:

[0018] The terminal outside the target vehicle is used to extract the movement amount of the preset feature in the initial panoramic image, and the actual movement amount of the target vehicle is calculated according to the movement amount.

[0019] In a specific embodiment, the acquiring the state data of the target vehicle at the current moment and correcting the state data according to the actual movement amount comprises:

[0020] Using a preset sensor to obtain state data of the target vehicle at the current moment, the state data including wheel-related parameters;

[0021] The state data is corrected according to the pre-established relationship between the wheel-related parameters and the vehicle movement amount and the actual movement amount.

[0022] In a specific embodiment, the relationship between the wheel-related parameters and the vehicle motion amount includes fitting the pre-collected wheel-related parameters and the vehicle motion amount using at least one of Bayesian theorem, Kalman filtering principle or PID control method.

[0023] In a specific embodiment, performing vehicle chassis image matching on the first intermediate image according to the corrected state data to obtain the second intermediate image comprises:

[0024] Calculating the rotation angle and the offset of the target vehicle between two adjacent moments according to the corrected state data;

[0025] Establishing a world coordinate system, and obtaining a transformation relationship of the world coordinates of the vertices of the target vehicle area between two adjacent moments according to the rotation angle and the offset;

[0026] Obtaining the world coordinates of the vertices of the target vehicle region at the current moment at the previous moment according to the transformation relationship of the world coordinates, and obtaining the image coordinates of the vertices of the target vehicle region at the current moment in the image at the previous moment according to the relationship between the world coordinate system and the image coordinate system;

[0027] A corresponding panoramic image of the target vehicle area at the current moment is intercepted from the initial panoramic image at the previous moment according to the image coordinates, and the panoramic image of the target vehicle area at the current moment is mapped to the initial panoramic image at the current moment to obtain a second intermediate image.

[0028] In a specific embodiment, the method further comprises:

[0029] Repeat steps S100 to S400 for the target panoramic image obtained in step S500 to obtain a panoramic image displayed in real time.

[0030] In a specific embodiment, the preprocessing includes at least one of noise removal, image enhancement, image segmentation and correction.

[0031] In a second aspect, corresponding to the above-mentioned method for generating a target panoramic image, a device for generating a target panoramic image is also provided, the device comprising:

[0032] A preprocessing module, used to obtain an initial panoramic image at a current moment and preprocess the initial panoramic image to obtain a first intermediate image;

[0033] A calculation module, used for extracting the movement amount of the preset features in the initial panoramic image, and calculating the actual movement amount of the target vehicle according to the movement amount;

[0034] A correction module, used for obtaining the state data of the target vehicle at the current moment, and correcting the state data according to the actual movement amount;

[0035] A matching module, configured to perform vehicle chassis image matching on the first intermediate image according to the corrected state data to obtain a second intermediate image;

[0036] The fusion module is used to fuse the second intermediate image to obtain a target panoramic image including a vehicle chassis image.

[0037] In a third aspect, a computer system is also provided, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the method for generating the target panoramic image is implemented.

[0038] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the method for generating the target panoramic image is implemented.

[0039] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0040] The present application provides a method, device, computer system and storage medium for generating a target panoramic image, the method comprising obtaining an initial panoramic image at the current moment and preprocessing the initial panoramic image to obtain a first intermediate image; extracting the movement amount of a preset feature in the initial panoramic image, and calculating the actual movement amount of the target vehicle according to the movement amount; obtaining the state data of the target vehicle at the current moment, and correcting the state data according to the actual movement amount; matching the first intermediate image with the vehicle chassis image according to the corrected state data to obtain a second intermediate image; fusing the second intermediate image to obtain a target panoramic image containing the vehicle chassis image. The present application scheme can eliminate the problem of stitching image distortion caused by the mismatch between wheel speed and actual vehicle speed, thereby improving the accuracy and reliability of the stitching of the vehicle transparent chassis image.

[0041] Furthermore, the method, device, computer system and storage medium for generating a target panoramic image provided by the present application utilize a terminal outside the target vehicle to extract the movement of preset features in the initial panoramic image, calculate the actual movement of the target vehicle based on the movement, separate the calculations with heavy computational load from the on-board chip end and simulate them on the external terminal, thereby improving computing efficiency.

[0042] Not all products in this application need to have all the above effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 is a flow chart of a method for generating a target panoramic image provided by an embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of a vehicle motion model provided by the implementation of this application;

[0046] Figure 3 It is a schematic diagram of the computer-side fitting and the vehicle-side directly correcting the state data according to the actual movement amount;

[0047] Figure 4 is a schematic diagram of the structure of a device for generating a target panoramic image provided by an embodiment of the present application;

[0048] Figure 5 It is an architectural diagram of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] As described in the background technology, in the existing vehicle transparent chassis image stitching solutions, when the vehicle's own position has a large error in time or position during the stitching of the vehicle's bottom image and the panoramic image, the panoramic image and the vehicle's bottom image will be misaligned during the stitching, and the vehicle's bottom image itself will be distorted. Since it is difficult to measure the vehicle's own position directly by the vehicle itself, the vehicle's driving direction and speed can be used as an alternative to infer the vehicle's position. However, it is very expensive to directly install a sensor to measure the vehicle's speed on the vehicle. Therefore, the vehicle's speed is usually inferred by measuring the speed of the wheel speedometer, thereby indirectly inferring and calculating the vehicle's own position. Therefore, during the vehicle transparent chassis image stitching process, if the wheel speed sensor and the actual vehicle speed do not match, the stitched image will be distorted and misaligned, so that the stitched vehicle bottom pattern cannot provide valuable prompt information to the driver.

[0051] In response to the above problems, the present application proposes a new method for generating a target panoramic image based on image processing technology, which is used to solve the problem of distortion and misalignment of the stitched image due to the mismatch between the wheel speed and the actual vehicle speed when the vehicle transparent chassis image is stitched. Specifically, the present application provides a method for generating a target panoramic image based on image fusion, which is used to stitch the vehicle chassis images at different positions in the panoramic image according to the corrected vehicle sensor data, and can eliminate the problem of stitched image distortion caused by the mismatch between the wheel speed and the actual vehicle speed. At the same time, only the correction part with a small amount of calculation is run on the vehicle side to meet the real-time requirements of the software. The method for generating a target panoramic image provided by the present application can be applied to various types of vehicles and has broad application prospects.

[0052] The solution of the present application will be described in detail below in conjunction with the accompanying drawings and various embodiments.

[0053] Embodiment 1

[0054] Figure 1 is a flow chart of a method for generating a target panoramic image provided by an embodiment of the present application, with reference to Figure 1 As shown, the method mainly includes the following steps:

[0055] S100: Acquire an initial panoramic image at a current moment and preprocess the initial panoramic image to obtain a first intermediate image.

[0056] As an exemplary but not limiting explanation, in the embodiment of the present application, the initial panoramic image can be obtained by setting multiple cameras around the vehicle, such as wide-angle cameras (such as fisheye cameras). In specific implementation, at least one fisheye camera can be set in each of the four directions of the front, back, left, and right of the vehicle to collect image information around the vehicle in real time. Since the fisheye camera is a special lens with an ultra-wide angle, its structure imitates the imaging of fish eyes, and can independently achieve wide-angle shooting. The angle of view of the fisheye camera can even reach 180°, and it can monitor objects in a wide range of scenes. A fisheye camera is set in each direction around the vehicle, and a panoramic image around the vehicle can be collected by 4 fisheye cameras, thereby saving the number of cameras and reducing costs. It should be noted here that the camera in the embodiment of the present application is not limited to a wide-angle camera, and the camera can also be an ordinary camera. In specific implementation, the angle of view of the camera can be increased by increasing the number of ordinary cameras. For example, 2 or 3 cameras are set in each direction around the vehicle, and the panoramic image around the vehicle is collected by ordinary cameras. Although the number of cameras is increased, the image information collected by the ordinary camera is not deformed, and the collected image effect is better.

[0057] In a specific embodiment, the initial panoramic image includes at least image information and depth information of objects around the target vehicle, wherein the depth information is used to indicate the three-dimensional coordinate information of each point on the objects around the target vehicle. For the convenience of calculation, depth information in the field of computer vision usually refers to the distance of each point in space relative to the camera sensor. In the embodiment of the present application, the depth information of objects around the target vehicle refers to the distance of the three-dimensional coordinate points on the objects around the target vehicle from the camera sensor.

[0058] Exemplarily, after the camera collects image information of objects around the vehicle, it transmits the image information to an image processing device (such as a vehicle-mounted image processing device, etc.). The image processing device can obtain depth information of objects around the vehicle based on the image information. This process can refer to the existing image processing process and will not be described in detail here.

[0059] S200: extracting the movement amount of the preset features in the initial panoramic image, and calculating the actual movement amount of the target vehicle according to the movement amount.

[0060] Specifically, in the embodiments of the present application, there are no specific restrictions on the preset features, and users can set them according to actual needs.

[0061] Specifically, the actual movement amount of the target vehicle can be calculated using the vehicle motion model. As a preferred implementation, in the embodiment of the present application, the vehicle motion model is set so that the left and right rear wheels always keep moving forward, and the steering movement of the vehicle is reflected by the wheel speed difference between the left and right rear wheels. The movement of the vehicle includes pure rotation around the rotation center and uniform translational movement of all positions of the vehicle body.

[0062] S300: Acquire the state data of the target vehicle at the current moment, and correct the state data according to the actual movement amount.

[0063] Specifically, in the embodiment of the present application, the state data of the target vehicle includes but is not limited to wheel speed related data, tire rotation angle and angular velocity, etc. In the embodiment of the present application, it is proposed to correct the state data using the actual movement amount of the target vehicle to avoid problems such as the wheel speed related data in the state data not being consistent with the actual vehicle speed.

[0064] S400: Perform vehicle chassis image matching on the first intermediate image according to the corrected state data to obtain a second intermediate image.

[0065] Specifically, by inferring the position of the vehicle through the corrected state data of the target vehicle (i.e., performing vehicle chassis image matching on the first intermediate image), the display content of the spliced ​​underbody images at different positions and the surrounding panoramic images can be matched without obvious misalignment and distortion of the image itself.

[0066] S500: Fusing the second intermediate images obtained by matching at different positions to obtain a target panoramic image including a vehicle chassis image.

[0067] Specifically, the second intermediate image obtained by matching at different positions is fused with the vehicle body position and the panoramic image through a fusion algorithm to obtain a new image containing the vehicle chassis (ie, the target panoramic image).

[0068] As a preferred implementation, in the embodiment of the present application, the extracting the movement amount of the preset feature in the initial panoramic image and calculating the actual movement amount of the target vehicle according to the movement amount includes:

[0069] Calculating the movement amount of the preset features in the initial panoramic image based on a dense optical flow algorithm;

[0070] Obtaining a rotation center of the target vehicle according to the movement amount;

[0071] The target vehicle is subjected to motion decomposition according to the rotation center and an actual motion amount of the target vehicle is calculated.

[0072] Specifically, in the embodiment of the present application, it is preferred to calculate the movement amount of the preset features in the initial panoramic image based on the dense optical flow algorithm to obtain the rotation center of the vehicle, thereby performing vehicle motion decomposition and displacement calculation (i.e., the actual movement amount of the target vehicle, expressed as r 真 , Δx 真 , Δy 真 , Δφ 真 ). The advantage of this method is that it can obtain the real information of vehicle movement, but this method requires the use of complex algorithms and computing resources. Therefore, the true value of the vehicle speed can be solved by using the optical flow method in the computer, and compared and fitted with the wheel speed sensor, and the fitting scheme can be used as an abstract output result to optimize the algorithm of the vehicle chip.

[0073] The following example illustrates the above calculation process:

[0074] Assume that the translational motion rate of the target vehicle is v tran , the radius of rotation of the rotational motion is r turn , the angular velocity of rotation is ω turn , the vehicle angle is φ, and the speed of the left rear wheel is v RL , the speed of the right rear wheel is v RR , then:

[0075] v tran =min(v RL , v RR )

[0076]

[0077]

[0078] Δx=v tran Δtsinφ, Δy=v tran Δtcosφ, Δφ=ω turn Δt

[0079] Among them, Δx and Δy are the offsets of the target vehicle in the x and y directions respectively, Δφ is the rotation angle of the target vehicle, Track is the rear wheel track of the vehicle, and Δt is the cycle interval of the vehicle braking system signal.

[0080] Reference Figure 2 As shown, T1 represents the final position of the vehicle in the world coordinate system, T2 represents the position of the vehicle after rotation, T3 represents the initial position of the vehicle in the panoramic image coordinate system, and T4 represents the position of the vehicle after removing the translation motion.

[0081] It can be understood that in a three-dimensional coordinate system, the translation and rotation of the image can use corresponding operation matrices, where the translation matrix corresponds to the translation movement of the vehicle body, and the rotation matrix corresponds to the rotation movement of the vehicle body. z is the direction perpendicular to the plane where x and y are located (perpendicular to the ground). In the embodiment of the present application, z=0, Δz=0, indicating that the displacement of the target vehicle in the direction perpendicular to the ground is 0.

[0082] As an example, the translation matrix can be expressed as:

[0083]

[0084] Among them, x, y, z correspond to the translation of the target vehicle in each direction Δx, Δy, Δz

[0085] As an example, a rotation matrix can be expressed as:

[0086]

[0087] Among them, x, y, z correspond to the coordinates of the rotation center (r turn , 0, 0), φ is the body rotation angle Δφ.

[0088] Assume that the position of the target vehicle at the current time t is The position of the target vehicle at the next moment t+1 is but

[0089]

[0090] In addition, since the image is transformed in the panoramic coordinate system at the current moment t, the position of the vehicle chassis in the panoramic image coordinate system remains unchanged when updating to the next moment t+1. Therefore, the coordinates of each pixel point at moment t must be updated to the panoramic image coordinate system at moment t+1. The specific process can refer to the coordinate system conversion method in the prior art, which will not be described here one by one.

[0091] In a specific embodiment, the method further includes comparing the pixel size in the image with the actual size of the vehicle to obtain actual motion information parameters, and reading the image directly in front of the vehicle into the buffer each time by image rotation and other methods, and the buffer image also updates its internal information each time by the position information. Finally, the content of the panoramic coordinate system is converted into the image coordinate system for display on the screen.

[0092] As a preferred implementation, in the embodiment of the present application, the extracting the movement amount of the preset feature in the initial panoramic image and calculating the actual movement amount of the target vehicle according to the movement amount includes:

[0093] The terminal outside the target vehicle is used to extract the movement amount of the preset feature in the initial panoramic image, and the actual movement amount of the target vehicle is calculated according to the movement amount.

[0094] Specifically, in the embodiment of the present application, it is preferred to calculate the actual movement of the target vehicle based on the dense optical flow algorithm. Since the dense optical flow algorithm has a large computational load, in order to improve the computational efficiency, the dense optical flow method calculation with a large computational load is separated from the on-board chip end and simulated on a terminal outside the target vehicle (such as a computer). The processing results of the external terminal are then output back to the on-board chip in an abstract fitting manner to fit the sensor data, thereby achieving the purpose of improving computational efficiency.

[0095] As a preferred implementation, in the embodiment of the present application, the acquiring the state data of the target vehicle at the current moment and correcting the state data according to the actual movement amount comprises:

[0096] Using a preset sensor to obtain state data of the target vehicle at the current moment, the state data including wheel-related parameters;

[0097] The state data is corrected according to the pre-established relationship between the wheel-related parameters and the vehicle movement amount and the actual movement amount.

[0098] Specifically, the preset sensors include but are not limited to wheel speed sensors, and the acquired status data include but are not limited to: wheel speed related data (such as the left rear wheel speed v RL , the speed of the right rear wheel is v RR etc.), the tire rotation angle (such as the left rear tire rotation angle is cnt RL , the right rear tire turns by an angle of cnt RR etc.) and angular velocity Yaw, etc.

[0099] It can be understood that the relationship between the wheel-related parameters and the vehicle motion is obtained by fitting the collected wheel-related parameters and the vehicle motion in accordance with a preset algorithm. Figure 3 As shown, the process can be performed on the computer side (such as a terminal outside the target vehicle) or on the vehicle side (such as an on-board chip). Since the computational load of this process is large, performing the calculation on the computer side can improve the computational efficiency compared to performing it on the vehicle side.

[0100] As a preferred implementation manner, in an embodiment of the present application, the relationship between the wheel-related parameters and the vehicle's motion volume includes fitting the pre-collected wheel-related parameters and the vehicle's motion volume using at least one of the Bayesian theorem, the Kalman filter principle or the PID control method.

[0101] It should be noted here that the above fitting method is only an illustrative and non-restrictive description of the present application. In the embodiments of the present application, no specific limitation is made to the method used for fitting. Under the premise of not violating the inventive concept of the present application, any known data fitting method can be used in the present application.

[0102] As a preferred implementation manner, in the embodiment of the present application, performing vehicle chassis image matching on the first intermediate image according to the corrected state data to obtain the second intermediate image includes:

[0103] Calculating the rotation angle and the offset of the target vehicle between two adjacent moments according to the corrected state data;

[0104] Establishing a world coordinate system, and obtaining a transformation relationship of the world coordinates of the vertices of the target vehicle area between two adjacent moments according to the rotation angle and the offset;

[0105] Obtaining the world coordinates of the vertices of the target vehicle region at the current moment at the previous moment according to the transformation relationship of the world coordinates, and obtaining the image coordinates of the vertices of the target vehicle region at the current moment in the image at the previous moment according to the relationship between the world coordinate system and the image coordinate system;

[0106] A corresponding panoramic image of the target vehicle area at the current moment is intercepted from the initial panoramic image at the previous moment according to the image coordinates, and the panoramic image of the target vehicle area at the current moment is mapped to the initial panoramic image at the current moment to obtain a second intermediate image.

[0107] Specifically, the interval between two adjacent moments can be determined based on the video frame rate of the camera. The higher the frame rate, the better the real-time computing effect can be reflected. It is generally 25 to 30 fps, that is, the time interval between two moments is 33 to 40 ms. In a specific embodiment, the acquired state data includes the translational motion rate of the body of the target vehicle, v tran , the radius of rotation of the rotational motion is r turn , the angular velocity of rotation is ω turn , the vehicle angle is φ, and the speed of the left rear wheel is v RL , the speed of the right rear wheel is v RR , then the rotation angle of the vehicle center at the current moment relative to the previous moment is expressed as:

[0108]

[0109] Wherein, Track is the rear wheel track of the vehicle, and Δt is the cycle interval of the vehicle braking system signal.

[0110] The offset of the vehicle center at the current moment relative to the previous moment is expressed as:

[0111] Δx=v tran Δtsinφ

[0112] Δy=v tran Δtcosφ

[0113] Among them, Δx and Δy are the offsets of the target vehicle in the x and y directions respectively.

[0114] Specifically, the center point of the vehicle at the starting time of the two moments is used as the coordinate origin to establish a world coordinate system. The x-axis and y-axis of the world coordinate system are located in the plane where the ground is located, and the coordinate perpendicular to the ground is zero. According to the calculated rotation angle and offset of the target vehicle, the transformation relationship of the world coordinates of the vertices of the vehicle area between two adjacent moments is obtained. The transformation relationship is expressed by the coordinate transformation matrix as follows:

[0115]

[0116] Among them, ×′ and y′ represent the world coordinates of the vertices of the target vehicle area at the current moment, and × and y represent the world coordinates of the vertices of the target vehicle area at the previous moment. The world coordinates of the vertices of the vehicle area at the previous moment can be obtained based on the vehicle parameters combined with the world coordinates of the vehicle center at the previous moment, which will not be repeated here.

[0117] Furthermore, an image coordinate system is established with the upper left corner of the panoramic image displayed by the panoramic surround view system as the origin, with the direction toward the bottom of the image as the x-axis direction of the image coordinate system, and the direction toward the right of the image as the y-axis direction of the image coordinate system. The image coordinates of the vertices of the vehicle area at the current moment in the image at the previous moment are expressed as:

[0118]

[0119] Among them, x" and y" are the image coordinates of the vertices of the vehicle area at the current moment in the image at the previous moment, Ratio is the ratio of the image coordinates to the world coordinates, and h and w are the length and width of the panoramic image respectively.

[0120] As a preferred implementation, in the embodiment of the present application, the method further includes:

[0121] Repeat steps S100 to S400 for the target panoramic image obtained in step S500 to obtain a panoramic image displayed in real time.

[0122] Specifically, as the vehicle continues to move, the above steps S100-S400 are repeated for the fused new image, that is, different parts of the new image are spliced ​​according to the position of the moving vehicle body, so as to continuously update the panoramic and chassis images, and then obtain a complete panoramic image with no blind spots including the vehicle chassis image.

[0123] As a preferred implementation, in the embodiment of the present application, the preprocessing includes at least one of noise removal, image enhancement, image segmentation and correction.

[0124] Specifically, the acquired initial panoramic image is preprocessed to improve the image quality and the accuracy of the subsequent calculation results. In the embodiment of the present application, no specific limitation is made to the preprocessing, which includes but is not limited to noise removal, image enhancement, image segmentation and correction, etc.

[0125] Embodiment 2

[0126] Corresponding to the above-mentioned embodiment 1, the present application further provides a device for generating a target panoramic image, wherein in this embodiment, the same or similar contents as those of the above-mentioned embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 4 As shown, the device comprises:

[0127] A preprocessing module, used to obtain an initial panoramic image at a current moment and preprocess the initial panoramic image to obtain a first intermediate image;

[0128] A calculation module, used for extracting the movement amount of the preset features in the initial panoramic image, and calculating the actual movement amount of the target vehicle according to the movement amount;

[0129] A correction module, used for obtaining the state data of the target vehicle at the current moment, and correcting the state data according to the actual movement amount;

[0130] A matching module, configured to perform vehicle chassis image matching on the first intermediate image according to the corrected state data to obtain a second intermediate image;

[0131] The fusion module is used to fuse the second intermediate image to obtain a target panoramic image including a vehicle chassis image.

[0132] As a preferred implementation, in the embodiment of the present application, the calculation module is specifically used for:

[0133] Calculating the movement amount of the preset features in the initial panoramic image based on a dense optical flow algorithm;

[0134] Obtaining a rotation center of the target vehicle according to the movement amount;

[0135] The target vehicle is subjected to motion decomposition according to the rotation center and an actual motion amount of the target vehicle is calculated.

[0136] As a preferred implementation, in the embodiment of the present application, the calculation module is specifically used for:

[0137] The terminal outside the target vehicle is used to extract the movement amount of the preset feature in the initial panoramic image, and the actual movement amount of the target vehicle is calculated according to the movement amount.

[0138] As a preferred implementation, in the embodiment of the present application, the correction module is specifically used for:

[0139] Using a preset sensor to obtain state data of the target vehicle at the current moment, the state data including wheel-related parameters;

[0140] The state data is corrected according to the pre-established relationship between the wheel-related parameters and the vehicle movement amount and the actual movement amount.

[0141] As a preferred implementation manner, in an embodiment of the present application, the relationship between the wheel-related parameters and the vehicle's motion volume includes fitting the pre-collected wheel-related parameters and the vehicle's motion volume using at least one of the Bayesian theorem, the Kalman filter principle or the PID control method.

[0142] As a preferred implementation, in the embodiment of the present application, the matching module is specifically used for:

[0143] Calculating the rotation angle and the offset of the target vehicle between two adjacent moments according to the corrected state data;

[0144] Establishing a world coordinate system, and obtaining a transformation relationship of the world coordinates of the vertices of the target vehicle area between two adjacent moments according to the rotation angle and the offset;

[0145] Obtaining the world coordinates of the vertices of the target vehicle region at the current moment at the previous moment according to the transformation relationship of the world coordinates, and obtaining the image coordinates of the vertices of the target vehicle region at the current moment in the image at the previous moment according to the relationship between the world coordinate system and the image coordinate system;

[0146] A corresponding panoramic image of the target vehicle area at the current moment is intercepted from the initial panoramic image at the previous moment according to the image coordinates, and the panoramic image of the target vehicle area at the current moment is mapped to the initial panoramic image at the current moment to obtain a second intermediate image.

[0147] As a preferred implementation, in the embodiment of the present application, the preprocessing includes at least one of noise removal, image enhancement, image segmentation and correction.

[0148] Embodiment 3

[0149] Corresponding to the above-mentioned embodiments one and two, the present application also provides a computer system, including: a processor and a memory, the memory storing a computer program that can be run on the processor, and when the computer program is executed by the processor, the method for generating a target panoramic image provided by any one of the above-mentioned embodiments is executed.

[0150] in, Figure 5 The computer system 1500 is shown as an example, and may include a processor 1510, a video display adapter 1511, a disk drive 1512, an input / output interface 1513, a network interface 1514, and a memory 1520. The processor 1510, the video display adapter 1511, the disk drive 1512, the input / output interface 1513, the network interface 1514, and the memory 1520 may be communicatively connected via a communication bus 1530.

[0151] Among them, the processor 1510 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided by the present invention.

[0152] The memory 1520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1520 can store an operating system 1521 for controlling the operation of the electronic device and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device. In addition, a web browser 1523, a data storage management system 1524, and a device identification information processing system 1525, etc. can also be stored. The above-mentioned device identification information processing system 1525 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present invention. In short, when the technical solution provided by the present invention is implemented by software or firmware, the relevant program code is stored in the memory 1520 and is called and executed by the processor 1510.

[0153] The input / output interface 1513 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0154] The network interface 1514 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0155] The bus includes a path that transmits information between the various components of the device (eg, processor 1510, video display adapter 1511, disk drive 1512, input / output interface 1513, network interface 1514, and memory 1520).

[0156] In addition, the electronic device can also obtain information on specific collection conditions from the virtual resource object collection condition information database for use in condition judgment, etc.

[0157] It should be noted that, although the above device only shows a processor 1510, a video display adapter 1511, a disk drive 1512, an input / output interface 1513, a network interface 1514, a memory 1520, a bus, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present invention, and does not necessarily include all the components shown in the figure.

[0158] Embodiment 4

[0159] Corresponding to the above-mentioned embodiments one to three, the embodiments of the present application further provide a computer-readable storage medium, wherein, in this embodiment, the same or similar contents as those of the above-mentioned embodiments one to three can be referred to the above introduction and will not be repeated later.

[0160] The computer-readable storage medium stores a computer program thereon, and when the computer program is executed by the processor, the method for generating the target panoramic image as described above is implemented.

[0161] In some implementations, in the embodiments of the present application, when the computer program is executed by the processor, the steps corresponding to the method described in Embodiment 1 may also be implemented. Please refer to the detailed description in Embodiment 1 and no further details will be given here.

[0162] It can be seen from the above description of the implementation mode that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer system (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or some parts of the embodiments.

[0163] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0164] The technical solution provided by the present invention is described in detail above. The principle and implementation method of the present invention are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for generating a target panoramic image, characterized in that: The method comprises: S100: Acquire an initial panoramic image at a current moment and preprocess the initial panoramic image to obtain a first intermediate image; S200: extracting the movement amount of the preset feature in the initial panoramic image, and calculating the actual movement amount of the target vehicle according to the movement amount, wherein calculating the actual movement amount of the target vehicle according to the movement amount includes: Obtaining a rotation center of the target vehicle according to the movement amount; The target vehicle is subjected to motion decomposition according to the rotation center and the actual motion amount of the target vehicle is calculated, and the calculation formula includes: v tran =min(v RL ,v RR ) Δx=v tran Δt sinφ Δy=v tran Δt cosφ Dφ=ω turn Δt Among them, v tran is the translational motion rate of the target vehicle, r turn is the rotation radius of the rotational motion, ω turn is the angular velocity, φ is the vehicle angle, v RL is the left rear wheel speed, v RR is the speed of the right rear wheel, Δx and Δy are the offsets of the target vehicle in the x and y directions respectively, Δφ is the rotation angle of the target vehicle, Track is the rear wheelbase of the vehicle, and Δt is the cycle interval of the vehicle braking system signal; S300: Acquiring the state data of the target vehicle at the current moment, and correcting the state data according to the actual movement amount, including: Using a preset sensor to obtain state data of the target vehicle at the current moment, the state data including wheel-related parameters; Correcting the state data according to a pre-established relationship between wheel-related parameters and vehicle motion and the actual motion; S400: performing vehicle chassis image matching on the first intermediate image according to the corrected state data to obtain a second intermediate image; S500: Fusing the second intermediate images obtained by matching at different positions to obtain a target panoramic image including a vehicle chassis image.

2. The method for generating a target panoramic image according to claim 1, characterized in that: The step of extracting the movement amount of the preset feature in the initial panoramic image comprises: The movement amount of the preset features in the initial panoramic image is calculated based on a dense optical flow algorithm.

3. The method for generating a target panoramic image according to claim 1 or 2, characterized in that: The extracting the movement amount of the preset feature in the initial panoramic image and calculating the actual movement amount of the target vehicle according to the movement amount comprises: The terminal outside the target vehicle is used to extract the movement amount of the preset feature in the initial panoramic image, and the actual movement amount of the target vehicle is calculated according to the movement amount.

4. The method for generating a target panoramic image according to claim 1, characterized in that: The relationship between the wheel-related parameters and the vehicle motion amount is obtained by fitting the pre-collected wheel-related parameters and the vehicle motion amount using at least one of Bayesian theorem, Kalman filtering principle or PID control method.

5. The method for generating a target panoramic image according to claim 1 or 2, characterized in that: The performing vehicle chassis image matching on the first intermediate image according to the corrected state data to obtain the second intermediate image comprises: Calculating the rotation angle and the offset of the target vehicle between two adjacent moments according to the corrected state data; Establishing a world coordinate system, and obtaining a transformation relationship of the world coordinates of the vertices of the target vehicle area between two adjacent moments according to the rotation angle and the offset; Obtaining the world coordinates of the vertices of the target vehicle region at the current moment at the previous moment according to the transformation relationship of the world coordinates, and obtaining the image coordinates of the vertices of the target vehicle region at the current moment in the image at the previous moment according to the relationship between the world coordinate system and the image coordinate system; A corresponding panoramic image of the target vehicle area at the current moment is intercepted from the initial panoramic image at the previous moment according to the image coordinates, and the panoramic image of the target vehicle area at the current moment is mapped to the initial panoramic image at the current moment to obtain a second intermediate image.

6. The method for generating a target panoramic image according to claim 1 or 2, characterized in that: The method further comprises: Repeat steps S100 to S400 for the target panoramic image obtained in step S500 to obtain a panoramic image displayed in real time.

7. The method for generating a target panoramic image according to claim 1 or 2, characterized in that: The preprocessing includes at least one of noise removal, image enhancement, image segmentation and correction.

8. A device for generating a target panoramic image, characterized in that: The device comprises: A preprocessing module, used to obtain an initial panoramic image at a current moment and preprocess the initial panoramic image to obtain a first intermediate image; A calculation module is used to extract the movement amount of the preset feature in the initial panoramic image, and calculate the actual movement amount of the target vehicle according to the movement amount, wherein calculating the actual movement amount of the target vehicle according to the movement amount includes: Obtaining a rotation center of the target vehicle according to the movement amount; The target vehicle is subjected to motion decomposition according to the rotation center and the actual motion amount of the target vehicle is calculated, and the calculation formula includes: v tran =min(v RL ,v RR ) Δx=v tran Δt sinφ Δy=v tran Δt cosφ Δφ= turn Δt Among them, v tran is the translational motion rate of the target vehicle, r turn is the rotation radius of the rotational motion, turn is the angular velocity, φ is the vehicle angle, v RL is the left rear wheel speed, v RR is the speed of the right rear wheel, Δx and Δy are the offsets of the target vehicle in the x and y directions respectively, Δφ is the rotation angle of the target vehicle, Track is the rear wheelbase of the vehicle, and Δt is the cycle interval of the vehicle braking system signal; A correction module, used for obtaining the state data of the target vehicle at the current moment and correcting the state data according to the actual movement amount, comprises: Using a preset sensor to obtain state data of the target vehicle at the current moment, the state data including wheel-related parameters; Correcting the state data according to a pre-established relationship between wheel-related parameters and vehicle motion and the actual motion; A matching module, configured to perform vehicle chassis image matching on the first intermediate image according to the corrected state data to obtain a second intermediate image; The fusion module is used to fuse the second intermediate image to obtain a target panoramic image including a vehicle chassis image.

9. A computer system, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the computer program is executed by the processor, the method for generating a target panoramic image according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, characterized in that: When the computer program is executed, the method for generating a target panoramic image according to any one of claims 1 to 7 is implemented.

Images