Vehicle panoramic image generation method and device, electronic equipment and storage medium

Abstract

The vehicle panoramic image generation method and device, the electronic device and the storage medium provided by the embodiment of the present application relate to the technical field of image processing, can obtain a target image collected by a target camera of a target vehicle body unit, identify image position information of a target calibration object in the target image, calculate a first relative position of the target camera and the target calibration object according to the image position information of the target calibration object, determine a target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in a turning state according to the first relative position, and splice driving environment images collected by cameras on the first vehicle body unit and the second vehicle body unit by using the target included angle to obtain a panoramic image. By the method of the embodiment of the present application, the panoramic image of the vehicle with the vehicle body not on the same straight line when turning can be generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a vehicle panoramic image generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of China's economy, vehicles are used more and more. At present, a vehicle often forms a 360-degree panoramic image through calibration, distortion correction and image stitching fusion of multiple cameras installed around the vehicle, so as to observe the situation around the vehicle body, eliminate the blind area of the vehicle body and improve the safety of driving.

[0003] The applicant found that the current panoramic view uses four cameras installed around the vehicle body to complete stitching. However, for vehicles such as a tractor and trailer, when turning, the front and rear vehicle bodies are not on the same straight line, so it is difficult to determine the vehicle body angle when turning, which leads to difficulty in stitching. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a vehicle panoramic image generation method, device, electronic equipment and storage medium to generate a panoramic image for a vehicle whose vehicle bodies are not on the same straight line when turning. The specific technical solutions are as follows:

[0005] In a first aspect, the present application provides a vehicle panoramic image generation method applied to a vehicle including a first vehicle body unit and a second vehicle body unit, at least a first side and a second side of each vehicle body unit are respectively provided with a camera and a calibration object, and the first side and the second side are the left and right sides of the vehicle body along the driving direction of the vehicle;

[0006] The method comprises:

[0007] Obtaining a target image collected by a target camera of a target vehicle body unit, wherein the target camera refers to a camera on the inner side of the target vehicle body unit when the vehicle is in a turning state;

[0008] Identifying image position information of a target calibration object in the target image, wherein the target calibration object and the target camera are located in different vehicle body units;

[0009] According to the image position information of the target calibration object, calculating a first relative position of the target camera and the target calibration object;

[0010] According to the first relative position, determining a target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in a turning state;

[0011] The image stitching module stitches the driving environment images captured by the cameras on the first vehicle body unit and the second vehicle body unit to obtain a panoramic image by using the target included angle.

[0012] In a second aspect, the embodiment of the present application provides a vehicle panoramic image generation device, which is applied to a vehicle including a first vehicle body unit and a second vehicle body unit, at least a first side and a second side of each vehicle body unit are respectively provided with a camera and a calibration object, and the first side and the second side are left and right sides of the vehicle body along a driving direction of the vehicle.

[0013] The device includes:

[0014] An image acquisition module acquires a target image captured by a target camera of a target vehicle body unit, wherein the target camera refers to a camera on the inner side of the target vehicle body unit when the vehicle is in a turning state.

[0015] An image recognition module is configured to recognize image position information of a target calibration object in the target image, wherein the target calibration object and the target camera are located in different vehicle body units.

[0016] A position calculation module is configured to calculate a first relative position between the target camera and the target calibration object according to the image position information of the target calibration object.

[0017] An included angle calculation module is configured to determine a target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in a turning state according to the first relative position.

[0018] An image stitching module is configured to stitch driving environment images captured by the cameras on the first vehicle body unit and the second vehicle body unit to obtain a panoramic image by using the target included angle.

[0019] In another aspect, the embodiment of the present application provides an electronic device, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0020] The memory is configured to store a computer program.

[0021] The processor is configured to execute the program stored on the memory to implement any of the vehicle panoramic image generation methods.

[0022] In another aspect, the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement any of the vehicle panoramic image generation methods.

[0023] Another aspect of the embodiments of the present application also provides a computer program product containing instructions which, when executed on a computer, cause the computer to perform any of the vehicle panoramic image generation methods described above.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] The vehicle panoramic image generation method, device, electronic equipment and storage medium provided by the embodiments of the present application can obtain a target image collected by a target camera of a target vehicle body unit, wherein the target camera refers to a camera on the inner side of the target vehicle body unit when the vehicle is in a turning state; identify image position information of a target calibration object in the target image, wherein the target calibration object and the target camera are located in different vehicle body units; calculate a first relative position of the target camera and the target calibration object according to the image position information of the target calibration object; determine a target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in a turning state according to the first relative position; and splice the driving environment images collected by the cameras on the first vehicle body unit and the second vehicle body unit by using the target included angle to obtain a panoramic image. Through the method of the embodiments of the present application, the image position information of the target calibration object in the target image can be identified, so as to calculate the first relative position of the target camera and the target calibration object according to the identified image position information, determine the target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in a turning state according to the first relative position, splice the driving environment images collected by the cameras on the first vehicle body unit and the second vehicle body unit by using the target included angle, and obtain a panoramic image, so as to realize the generation of a panoramic image for a vehicle whose vehicle body is not on the same straight line when turning.

[0026] Of course, implementing any product or method of the present application does not necessarily require all the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0028] Figure 1 A flowchart of a vehicle panoramic image generation method provided for the examples of the present application;

[0029] Figure 2 A structure diagram of a trailer provided for the examples of the present application;

[0030] Figure 3 A schematic diagram of a camera coordinate system provided for the embodiments of the present application;

[0031] Figure 4 A schematic diagram of a calibration board arrangement around a vehicle body provided for the embodiments of the present application;

[0032] Figure 5 A schematic diagram of stitching an overhead view of a tractor provided for the embodiments of the present application;

[0033] Figure 6 A schematic diagram of rotating a stitched image of a vehicle body unit provided for the embodiments of the present application;

[0034] Figure 7 Another schematic diagram of a vehicle panoramic image generation method provided for the embodiments of the present application;

[0035] Figure 8 A schematic diagram of stitching an overhead view of a vehicle provided for the embodiments of the present application;

[0036] Figure 9 A schematic diagram of determining a weight of a pixel provided for the embodiments of the present application;

[0037] Figure 10 An example diagram of determining a weight of a pixel provided for the embodiments of the present application;

[0038] Figure 11 A schematic diagram of obtaining an image position provided for the embodiments of the present application;

[0039] Figure 12 A schematic diagram of stitching a stitched image of each vehicle body unit provided for the embodiments of the present application;

[0040] Figure 13 A schematic diagram of a vehicle panoramic image generation method provided for the embodiments of the present application;

[0041] Figure 14 A schematic diagram of an electronic device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application are within the scope of protection of the present application.

[0043] First, the professional terms that can be used in the embodiments of the present application are explained:

[0044] Surround view system is through the installation in the car body four fisheye cameras around real-time collection vehicle around the image, after distortion correction, view conversion, graphics splicing, rendering output processing, eventually form a complete car body around the panoramic bird's-eye view, is a kind of auxiliary parking system.

[0045] CAN (Controller Area Network) is a kind of field bus.

[0046] The trailer is composed of the front tractor and the rear carriage, the front tractor drags the rear carriage, and the two carriages are connected in a non-rigid body and can rotate.

[0047] The camera internal parameter is the parameter related to the camera itself, such as the focal length, the optical center position, the distortion coefficient and the like.

[0048] The camera external parameter is the parameter of the camera in the world coordinate system, such as the position (X, Y, Z) of the camera, the rotation direction (pitch angle, roll angle, yaw angle) and the like.

[0049] Fusion refers to the fact that when two images are spliced together, obvious seams are easy to appear, and the fusion strategy used to eliminate the seams between the two images can make the two images into one and eliminate the seams.

[0050] The first aspect of the embodiment of the application provides a vehicle panoramic image generation method, which is applied to a vehicle including a first vehicle body unit and a second vehicle body unit, a camera and a calibration object are arranged on at least a first side and a second side of each vehicle body unit respectively, and the first side and the second side are left and right sides of the vehicle body along the driving direction of the vehicle. Figure 1 , Figure 1 A flowchart of the vehicle panoramic image generation method provided by the embodiment of the application is shown in FIG. 1.

[0051] In step S11, a target image collected by a target camera of a target vehicle body unit is acquired.

[0052] The target camera can refer to a camera installed on the inner side of the target vehicle body unit when the vehicle is in a turning state, for example, a camera installed on the left side of the vehicle when the vehicle turns left, or a camera installed on the right side of the vehicle when the vehicle turns right. In the embodiments of the present application, the target vehicle body unit can include a first vehicle body unit and / or a second vehicle body unit when the vehicle is in a turning state. Correspondingly, the target image captured by the target camera of the target vehicle body unit includes at least one of the following: a target image captured by the target camera of the first vehicle body unit; wherein the target image includes at least a second calibration object on the inner side of the second vehicle body unit when the vehicle is in a turning state; a target image captured by the target camera of the second vehicle body unit; wherein the target image includes at least a first calibration object on the inner side of the first vehicle body unit when the vehicle is in a turning state. The first vehicle body unit can be the vehicle body unit where the vehicle head is located, and the second vehicle body unit can be the vehicle body unit where the vehicle tail is located. Alternatively, the first vehicle body unit can be the vehicle body unit where the vehicle tail is located, and the second vehicle body unit can be the vehicle body unit where the vehicle head is located. The present application does not limit this. When capturing the target image of the target vehicle body unit, the camera installed on the left side of the vehicle body unit where the vehicle head is located can be used to capture the image of the calibration object installed on the left side of the vehicle body unit where the vehicle tail is located when the vehicle turns left, or the camera installed on the left side of the vehicle body unit where the vehicle tail is located can be used to capture the image of the calibration object installed on the left side of the vehicle body unit where the vehicle head is located.

[0053] In actual use, the vehicle in the embodiments of the present application can include multiple vehicle body units, and each vehicle body unit has at least one camera and one calibration object installed on both sides of the vehicle body. For example, referring to Figure 2 , the trailer is composed of a tractor and a carriage, and at least one camera and one calibration board are installed on both sides of the tractor and the carriage. In the embodiments of the present application, in order to realize panoramic image, there can be sufficient overlapping area between adjacent cameras, and the definition should be high. Alternatively, the camera in the embodiments of the present application can be a camera with a horizontal field of view greater than or equal to 180 degrees, so as to ensure sufficient overlapping area between adjacent cameras. Specifically, it can be a fisheye camera, etc. Since the central definition of the camera is generally high, the periphery is blurred. In order to ensure the definition, when installing the camera, the peripheral area of the vehicle body should be ensured to fall as much as possible in the center of the camera field of view. The calibration object in the embodiments of the present application can be a calibration block or a calibration board, etc., which is an object for calibrating the position.

[0054] The method of the embodiment of the present application is applied to a smart terminal, and the panoramic image can be generated by the smart terminal. Specifically, the smart terminal can be a terminal device installed on a vehicle or independent of the vehicle. For example, when the vehicle collects an image, the panoramic image can be synthesized by the processor of the vehicle or sent to the backend for synthesis, and then the synthesized panoramic image is fed back to the vehicle and displayed on the display device of the vehicle.

[0055] In step S12, the image position information of the target calibration object in the target image is recognized.

[0056] In the embodiment of the present application, the camera and the calibration object can be installed on both sides of the first vehicle body unit and the second vehicle body unit. For example, referring to Figure 2 The images captured by the camera 2 and the camera 4 can see the calibration board 1 and the calibration board 3, and the images captured by the camera 3 and the camera 5 can see the calibration board 2 and the calibration board 4. The camera 1 is a camera installed on the head of the tractor, and the camera 6 is a camera installed on the tail of the vehicle. Since the outside camera cannot recognize the other vehicle camera when the vehicle turns, the function will be disabled. Therefore, in actual use, the calibration boards are installed on the left and right sides, and different calibration boards are recognized according to different turning conditions of the vehicle, so that the angle between the front vehicle and the rear vehicle can be calculated in any case, and the left and right sides can always complete the calibration, thereby ensuring accurate splicing. Since the camera installed on the outside of the vehicle body may not be able to shoot the calibration object installed on the vehicle body when the vehicle turns, the position information of the calibration object installed on each vehicle body unit inside the turning direction of the vehicle can be recognized when the position information of the calibration object in the image captured by each camera is recognized. In actual use, the steering wheel signal, such as the can signal, can be read in real time, and when the turning angle is greater than a certain angle, it can be judged that the vehicle is turning, and the image captured by the camera inside the turning direction is selected for calibration object recognition.

[0057] In the embodiments of the present application, the image position information of the target calibration object in the target image (i.e., the position information of the target calibration object in the target image) can be used to identify the image position information of the second calibration object inside the second vehicle body unit in the target image collected by the target camera of the first vehicle body unit; and / or, the image position information of the first calibration object inside the first vehicle body unit in the target image collected by the target camera of the second vehicle body unit. Wherein, the target calibration object and the target camera are located in different vehicle body units, each side of the vehicle includes at least one camera and at least one calibration object, and when one side includes only one camera or calibration object, the camera and the calibration object are located on different vehicle body units. And in the turning state, the target camera can collect the image of the target calibration object. For example, the position information of the calibration object installed on the left side of the vehicle body unit of the tail in the image can be identified by the camera installed on the left side of the vehicle body unit of the head, or the position information of the calibration object installed on the left side of the vehicle body unit of the head in the image can be identified by the camera installed on the left side of the vehicle body unit of the tail.

[0058] In step S13, the first relative position between the target camera and the target calibration object is calculated according to the image position information of the target calibration object.

[0059] In the embodiments of the present application, the first relative position between the target camera and the target calibration object can be calculated according to the image position information of the target calibration object, which can include: calculating the first predicted coordinate position of the second calibration object in the first camera coordinate system according to the image position information of the second calibration object; creating a first covariance matrix according to the first predicted coordinate position, and determining the eigenvector corresponding to the minimum eigenvalue of the first covariance matrix and the coordinate mean corresponding to the first covariance matrix; calculating the first coordinate mapping relationship between the image coordinate system and the first camera coordinate system according to the eigenvector corresponding to the minimum eigenvalue of the first covariance matrix and the coordinate mean corresponding to the first covariance matrix; and calculating the first relative position between the target camera of the first vehicle body unit and the second calibration object inside the second vehicle body unit according to the image position information of the second calibration object and the first coordinate mapping relationship.

[0060] And / or,

[0061] According to the image position information of the first calibration object, a second predicted coordinate position of the first calibration object in a second camera coordinate system is calculated; a second covariance matrix is created according to the second predicted coordinate position, and a feature vector corresponding to a minimum eigenvalue of the second covariance matrix and a coordinate mean value corresponding to the second covariance matrix are determined; a second coordinate mapping relationship between the first calibration object in the image coordinate system and the second camera coordinate system is calculated according to the feature vector corresponding to the minimum eigenvalue of the second covariance matrix and the coordinate mean value corresponding to the second covariance matrix; and a first relative position of the first calibration object inside the first vehicle body unit from the target camera of the second vehicle body unit is calculated according to the image position information of the first calibration object and the second coordinate mapping relationship.

[0062] In the embodiments of the present application, the second camera coordinate system and the first camera coordinate system are used to distinguish the coordinate systems centered on the cameras installed on different vehicle body units. The first camera coordinate system can be a coordinate system with the camera installed on the first vehicle body unit as the coordinate center, and the second camera coordinate system can be a coordinate system with the camera installed on the second vehicle body unit as the coordinate center. The first predicted coordinates can include a plurality of predicted three-dimensional coordinates. The first covariance matrix is created according to the first predicted coordinate position, and the covariance of the plurality of predicted three-dimensional coordinates is calculated, and then the covariance matrix is constructed according to the plurality of predicted three-dimensional coordinates. For example, referring to Figure 3 , the predicted coordinate position of the calibration object in the camera coordinate system is p(x, y), the covariance matrix is created according to the predicted coordinate position, and the feature vector corresponding to the minimum eigenvalue of the covariance matrix and the coordinate mean value corresponding to the covariance matrix are determined; the coordinate mapping relationship between the calibration object in the image coordinate system and the camera coordinate system is calculated according to the feature vector corresponding to the minimum eigenvalue of the covariance matrix and the coordinate mean value corresponding to the covariance matrix, so that the relative position of the calibration object inside the vehicle body unit from the target camera of the vehicle body unit is calculated according to the image position information of the calibration object and the coordinate mapping relationship; and the predicted three-dimensional coordinates P(x w , y w , z w ) are obtained according to the relative position. Specifically, the predicted three-dimensional coordinates are (x, y, z), and the constructed covariance matrix is C;

[0063]

[0064] wherein, is the mean value of the horizontal coordinate X in the predicted three-dimensional coordinates, is the mean value of the vertical coordinate Y in the predicted three-dimensional coordinates, X i and Y iwherein the two coordinate values represent a coordinate point in the predicted three-dimensional coordinate, subscript i is used to represent different predicted three-dimensional coordinates, and n is the total number of the predicted three-dimensional coordinates. The specific calculation of the covariance matrix can refer to the existing principle, and the embodiments of the present application are not limited specifically.

[0065] The eigenvector corresponding to the minimum eigenvalue of the covariance matrix can be calculated by a matrix solution scheme, the coordinate mean corresponding to the second covariance matrix can be calculated, the coordinate means of the plurality of predicted three-dimensional coordinates are calculated, and then the mapping relationship between the image coordinate system and the camera coordinate system with the camera as the coordinate origin is obtained through iterative calculation according to the eigenvector corresponding to the minimum eigenvalue and the coordinate mean. Finally, the relative position of the camera and the calibration object is calculated according to the mapping relationship and the image position information. The method for calculating the first relative position of the target camera of the first vehicle body unit and the second calibration object inside the second vehicle body unit, and the first relative position of the target camera of the second vehicle body unit and the first calibration object inside the first vehicle body unit are similar, and can be referred to in this step.

[0066] For example, referring to Figure 4 The calibration board 1-10 is placed around the vehicle body to ensure that the images captured by the two adjacent cameras contain a unified marker. The external parameter calibration can include the following steps: fisheye image acquisition, distortion correction, corner point recognition, external parameter calculation, etc. The fisheye image acquisition can obtain the image captured by the fisheye camera, the distortion correction can correct the distortion of the image captured by the fisheye camera, the corner point recognition can recognize the preset object in the image to obtain the position information of the preset object, and the external parameter calculation can calculate the external parameter of the camera according to the position information obtained by the corner point recognition. Through the external parameter calibration, the mapping relationship from the world coordinate with the center of the vehicle body as the coordinate origin to the fisheye image can be found, and the camera imaging can realize the coordinate conversion between the world coordinate system, the camera coordinate system, the image coordinate system, the pixel coordinate system and other coordinate systems, and the internal and external parameters of the camera can be obtained to complete the coordinate system relationship conversion.

[0067] The world coordinate system in the embodiments of the present application can describe a certain object coordinate in the world coordinate, such as establishing a world coordinate system with a calibration board or taking a vehicle coordinate system as a world coordinate system. The camera coordinate system represents that the optical center is the origin, the optical axis is the Zc axis, and the Xc axis and the Yc axis are parallel to the imaging plane x axis and the y axis, respectively. The image coordinate system is a 2D coordinate system of the imaging plane, the origin is the intersection of the optical axis and the imaging plane, and the x axis and the y axis are parallel to the Xc axis and the Yc axis of the camera coordinate system, respectively. The pixel coordinate system can refer to the coordinate system of the finally presented image, the upper left corner is the origin, and the two axes can be parallel to the image coordinate system, and the unit can be pixels. When the world coordinate system is established with the vehicle body center as the origin, the front of the vehicle head and the right side of the vehicle can be the positive directions of the Xc axis and the Yc axis, respectively, and the calibration board is fixedly placed, and the world coordinate position of the calibration board is known. The position of the camera in the world coordinate system is calculated. In the world coordinate system in which the marker is located, after the covariance matrix of the marker is constructed, the eigenvector corresponding to the minimum eigenvalue of the covariance matrix and the coordinate mean value of the covariance matrix are calculated; the initial rotation matrix R of the marker is obtained by transformation using the eigenvector and the coordinate mean value; then, the initial translation vector T of the marker is calculated according to the initial rotation matrix R and the coordinate mean value; the initial rotation matrix R and the initial translation vector T are iteratively optimized by using the re-projection error between the coordinates of the marker in the calibration image and the coordinates of the marker in the world coordinate system in which the marker is located, to obtain the mapping relationship between the coordinates of the marker in the calibration image and the coordinates of the marker in the world coordinate system in which the marker is located; and the position relationship between the vehicle-mounted camera and the marker is obtained according to the mapping relationship and the intrinsic parameters of the vehicle-mounted camera.

[0068] In step S14, the target included angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle is determined according to the first relative position.

[0069] In the embodiments of the present application, the target included angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle is determined according to the first relative position. The first rotation angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle can be determined according to the first relative position of the target camera of the first vehicle body unit and the second calibration object on the inner side of the second vehicle body unit, and the first rotation angle is taken as the target included angle. Or, the second rotation angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle is determined according to the first relative position of the target camera of the second vehicle body unit and the first calibration object on the inner side of the first vehicle body unit, and the second rotation angle is taken as the target included angle. Or, the first rotation angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle is determined according to the first relative position of the target camera of the first vehicle body unit and the second calibration object on the inner side of the second vehicle body unit, and the second rotation angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle is determined according to the first relative position of the target camera of the second vehicle body unit and the first calibration object on the inner side of the first vehicle body unit, and then the average of the first rotation angle and the second rotation angle is determined as the target included angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle.

[0070] Specifically, the target included angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle is determined according to the first relative position, which can include: calculating the first coordinate position of the preset rotation point, the second coordinate position corresponding to the target camera in the non-turning state of the vehicle, and the third coordinate position corresponding to the target camera in the turning state of the vehicle in the preset coordinate system according to the first relative position, the second relative position of the target camera and the target calibration object in the non-turning state of the vehicle, and the third relative position of the target calibration object and the preset rotation point in the non-turning state of the vehicle; wherein the preset rotation point is the rotation center of the first vehicle body unit and the second vehicle body unit; calculating the included angle between the line connecting the first coordinate position and the second coordinate position and the line connecting the first coordinate position and the third coordinate position to obtain the target included angle. Wherein, the above-mentioned preset coordinate system can be a world coordinate system, and specifically, the preset coordinate system can be determined according to the calculation requirement, which is not limited in the embodiments of the present application. For example, the position of the target calibration object can be calibrated in advance, such as placing a calibration board around the vehicle body in the stationary state to obtain the second relative position of the target camera and the target calibration object in the non-turning state of the vehicle, and the third relative position of the target calibration object and the preset rotation point in the non-turning state of the vehicle, for example, taking a trailer as an example, placing a calibration board around the vehicle, and the placement position can be as follows: Figure 5As shown, the front camera can capture images of the calibration board 1 and the calibration board 2, the left front camera can capture images of the calibration board 1, the calibration board 10 and the calibration board 9, and the right front camera can capture images of the calibration board 2, the calibration board 3 and the calibration board 4, so as to ensure that the images captured by two adjacent cameras contain a unified marker, so as to calibrate according to the same marker, and obtain the second relative position of the target camera and the target marker in the non-turning state of the vehicle, and the third relative position of the target marker and the preset rotation point in the non-turning state of the vehicle. Referring to Figure 6 , the dashed line in the figure corresponds to the tractor before turning, and the solid line corresponds to the tractor after turning. The small circles in the figure represent the cameras, and the small squares represent the calibration boards. Therefore, when determining the target angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle, after obtaining the first relative position, the second relative position of the target camera and the target marker in the non-turning state of the vehicle and the third relative position of the target marker and the preset rotation point in the non-turning state of the vehicle obtained by the pre-calibration can be used to calculate the target angle.

[0071] In step S15, the target angle is used to splice the driving environment images captured by the cameras on the first vehicle body unit and the second vehicle body unit to obtain a panoramic image.

[0072] The driving environment image refers to an image of the driving environment captured during vehicle driving. For example, the spliced images of each vehicle body unit can be spliced according to the angle between each vehicle body unit and the adjacent vehicle body unit. The spliced images of each vehicle body unit can be rotated according to the calculated angle, and then the rotated spliced images can be spliced to obtain a panoramic image.

[0073] At present, in the prior art, when generating a panoramic image of a vehicle, the angle between the front and rear vehicle body units is obtained by a sensor, and then the images captured by the cameras are spliced according to the obtained angle to obtain a panoramic image of the vehicle. However, when the angle between the front and rear vehicle body units is captured by the sensor, the sensor may fail or have errors, resulting in a situation where the obtained angle between the front and rear vehicle body units is different from the actual angle. Through the method of the present application, the images of the calibration markers installed on another vehicle body unit can be captured by the cameras installed on the vehicle body, the angle between the front and rear vehicle body units can be calculated by recognizing the position of the calibration markers in the images, and the captured images can be spliced by the calculated angle to obtain a panoramic image. Since the angle in the present application is calculated from the actually captured images, the angle can be ensured to conform to the actual situation, so that the captured images can be spliced by the angle to obtain a panoramic image that conforms to the actual situation.

[0074] It can be seen that, by the method of the embodiment of the present application, the image position information of the target calibration object in the target image can be recognized, so that the first relative position of the target camera and the target calibration object is calculated according to the recognized image position information, and the target angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in the turning state is determined according to the first relative position. The driving environment images captured by the cameras on the first vehicle body unit and the second vehicle body unit are spliced by using the target angle, and a panoramic image is obtained, so that the panoramic image of the vehicle whose vehicle body is not in the same straight line when turning is generated.

[0075] Optionally, referring to Figure 7 The method further includes:

[0076] Step S71, the images captured by the cameras on the target vehicle body unit are subjected to first overlapping region recognition.

[0077] If the target vehicle body unit is the first vehicle body unit, and the first vehicle body unit is the vehicle body unit where the vehicle head is located, the first overlapping region includes: the overlapping region of the third image captured by the camera at the vehicle head of the first vehicle body unit and the fourth image captured by the camera on the first side, and / or the overlapping region of the third image captured by the camera at the vehicle head of the first vehicle body unit and the fourth image captured by the camera on the second side; or, if the target vehicle body unit is the second vehicle body unit, and the second vehicle body unit is the vehicle body unit where the vehicle tail is located, the first overlapping region includes: the overlapping region of the third image captured by the camera at the vehicle tail of the second vehicle body unit and the fourth image captured by the camera on the first side, and / or the overlapping region of the third image captured by the camera at the vehicle tail of the second vehicle body unit and the fourth image captured by the camera on the second side.

[0078] Step S72, for any pixel point in the first overlapping region of the third image and the fourth image with the overlapping region, the weight of the first pixel value corresponding to the pixel point in the third image and the weight of the second pixel value corresponding to the pixel point in the fourth image are calculated according to the position of the pixel point in the first overlapping region; if the position of the pixel point in the first overlapping region is closer to the third image, the weight of the first pixel value corresponding to the pixel point in the third image is greater, and if the position of the pixel point in the first overlapping region is closer to the fourth image, the weight of the second pixel value corresponding to the pixel point in the fourth image is greater.

[0079] Step S73, the target pixel value of the pixel point corresponding to the first pixel value and the second pixel value is determined according to the weight of the first pixel value and the weight of the second pixel value.

[0080] Step S74, according to the target pixel value of each pixel point in the first overlap region obtained by calculation, the third image and the fourth image are spliced to obtain the driving environment image corresponding to the target vehicle body unit.

[0081] Specifically, referring to Figure 8 , the rear camera can be collecting the images of the calibration board 6 and the calibration board 7, the left rear camera can be collecting the images of the calibration board 7, the calibration board 8 and the calibration board 9, and the right rear camera can be collecting the images of the calibration board 4, the calibration board 5 and the calibration board 6. Thus, when the first vehicle body unit is a tractor and the second vehicle body unit is a trailer, the overlap region can include: the front camera sub-top view of the tractor, and the overlap region of the left front camera sub-top view and the right front camera sub-top view, respectively; the rear camera sub-top view of the trailer, and the overlap region of the left rear camera sub-top view and the right rear camera sub-top view, respectively.

[0082] Optionally, the first overlap region includes a first target boundary and a second target boundary, and the first target boundary and the second target boundary intersect; the first target boundary and the second target boundary include at least one of the following cases:

[0083] (a) in the case that the first overlap region includes the overlap region of the third image collected by the camera at the front of the first vehicle body unit and the fourth image collected by the camera on the first side: if the first target boundary is the lower boundary of the first overlap region in the image height direction, then the second target boundary is the right boundary of the first overlap region in the image width direction; or, if the first target boundary is the left boundary of the first overlap region in the image width direction, then the second target boundary is the upper boundary of the first overlap region in the image height direction;

[0084] (b) in the case that the first overlap region includes the overlap region of the third image collected by the camera at the front of the first vehicle body unit and the fourth image collected by the camera on the second side: if the first target boundary is the lower boundary of the first overlap region in the image height direction, then the second target boundary is the left boundary of the first overlap region in the image width direction; or, if the first target boundary is the right boundary of the first overlap region in the image width direction, then the second target boundary is the upper boundary of the first overlap region in the image height direction;

[0085] (c) in the case that the first overlap region includes the overlap region of the third image collected by the camera at the rear of the second vehicle body unit and the fourth image collected by the camera on the first side: if the first target boundary is the upper boundary of the first overlap region in the image height direction, then the second target boundary is the right boundary of the first overlap region in the image width direction; or, if the first target boundary is the left boundary of the first overlap region in the image width direction, then the second target boundary is the lower boundary of the first overlap region in the image height direction;

[0086] (d) in the case that the first overlapping region comprises an overlapping region of a third image captured by a camera at a rear of the second vehicle body unit and a fourth image captured by a camera of the second side face: if the first target boundary is an upper boundary of the first overlapping region in the image height direction, the second target boundary is a left boundary of the first overlapping region in the image width direction; or, if the first target boundary is a right boundary of the first overlapping region in the image width direction, the second target boundary is a lower boundary of the first overlapping region in the image height direction.

[0087] By the method of the embodiments of the present application, the weight corresponding to a pixel point in an image can be calculated according to the position of the pixel point in the overlapping region, so as to determine the pixel value of each pixel point in the overlapping region according to the calculated weight, thereby realizing the splicing of the image and obtaining the driving environment image.

[0088] Optionally, referring to Figure 9 , the image height direction is a direction perpendicular to the image width direction, and for any pixel point in the first overlapping region of the third image and the fourth image, the weight of a first pixel value corresponding to the pixel point in the third image and the weight of a second pixel value corresponding to the pixel point in the fourth image are calculated according to the position of the pixel point in the first overlapping region, comprising:

[0089] Step S721, identifying the coordinates of the intersection of the first target boundary and the second target boundary to obtain a target origin;

[0090] Step S722, for any pixel point in the first overlapping region, the first angle between the line connecting the pixel point and the target origin and the first target boundary is calculated, and the second angle between the line connecting the pixel point and the target origin and the second target boundary is calculated according to the position of the pixel point in the first overlapping region.

[0091] Step S723, the weight of the first pixel value corresponding to the pixel point in the third image is determined according to the first angle, and the weight of the second pixel value corresponding to the pixel point in the fourth image is determined according to the second angle.

[0092] The first angle and the second angle in the embodiments of the present application are only used to distinguish different angles. The first target boundary and the second target boundary are only used to distinguish different boundaries. Optionally, the target pixel value of the pixel point corresponding to the first pixel value and the second pixel value is determined according to the weight of the first pixel value and the weight of the second pixel value, comprising: the target pixel value P of the pixel point corresponding to the first pixel value and the second pixel value is determined according to the weight of the first pixel value and the weight of the second pixel value through the following preset formula:

[0093]

[0094] wherein, a is the first angle, b is the second angle, m is the first pixel value, n is the second pixel value, P is the target pixel value of the pixel point in the first overlapping area, is the weight corresponding to the first pixel value, is the weight corresponding to the second pixel value. For example, referring to Figure 10 For the pixel point A in the overlapping area, the first target boundary is the OB side, and the second target boundary is the OA side, then ∠AOB is the first angle, and ∠AOC is the second angle.

[0095] By the method of the embodiments of the present application, when calculating the weight of the pixel point, the closer the pixel point is to a certain image, the greater the weight of the pixel point corresponding to the image, and the corresponding weight is obtained, so that when the corresponding target pixel value is calculated according to the weight and the image is spliced, the smooth transition of the final overlapping area and the image can be ensured, thereby improving the image quality of the panoramic image.

[0096] Optionally, referring to Figure 11 The target angle is used to splice the driving environment images collected by the cameras on the first vehicle body unit and the second vehicle body unit to obtain a panoramic image, comprising:

[0097] In step S151, the target angle is used to perform a translation transformation on the driving environment images collected by the cameras on the first vehicle body unit and / or the second vehicle body unit;

[0098] In step S152, the second overlapping area in the driving environment images of the corresponding first vehicle body unit and second vehicle body unit after the translation transformation operation is identified;

[0099] In step S153, for any pixel point in the second overlapping area, the weight of the third pixel value corresponding to the pixel point in the driving environment image of the first vehicle body unit after the translation transformation operation is calculated according to the position of the pixel point in the second overlapping area, and the weight of the fourth pixel value corresponding to the pixel point in the driving environment image of the second vehicle body unit after the translation transformation operation is calculated; wherein, the closer the position of the pixel point in the second overlapping area to the driving environment image of the first vehicle body unit, the greater the weight of the third pixel value corresponding to the pixel point, and the closer the position of the pixel point in the second overlapping area to the driving environment image of the second vehicle body unit, the greater the weight of the fourth pixel value corresponding to the pixel point;

[0100] In step S154, the target pixel value of the pixel point corresponding to the third pixel value and the fourth pixel value is determined according to the weight of the third pixel value and the weight of the fourth pixel value;

[0101] In step S155, the target pixel value of each pixel point in the second overlap region is calculated, and the driving environment images of the first vehicle body unit and the second vehicle body unit after the translation transformation operation are spliced to obtain a panoramic image of the vehicle.

[0102] Optionally, the second overlap region comprises a third target boundary and a fourth target boundary, the third target boundary is an upper boundary of the second overlap region in an image height direction, the fourth target boundary is a lower boundary of the second overlap region in the image height direction, the image height direction is perpendicular to the image width direction; the first vehicle body unit is a vehicle head body unit, the second vehicle body unit is a vehicle tail body unit, the upper boundary of the driving environment image of the first vehicle body unit in the image height direction is higher than the third target boundary, and the lower boundary of the driving environment image of the second vehicle body unit in the image height direction is lower than the fourth target boundary.

[0103] Optionally, for any pixel point in the second overlap region, the weight of the third pixel value corresponding to the pixel point in the driving environment image of the first vehicle body unit after the translation transformation operation and the weight of the fourth pixel value corresponding to the pixel point in the driving environment image of the second vehicle body unit after the translation transformation operation are calculated according to the position of the pixel point in the second overlap region, comprising: for any pixel point in the second overlap region, identifying a first distance of the pixel point from the third target boundary and a second distance of the pixel point from the fourth target boundary; according to the first distance and the second distance, determining the weight of the fourth pixel value corresponding to the pixel point in the driving environment image of the second vehicle body unit after the translation transformation operation and the weight of the third pixel value corresponding to the pixel point in the driving environment image of the first vehicle body unit after the translation transformation operation, wherein the value of the first distance is directly proportional to the value of the weight of the fourth pixel value, and the value of the second distance is directly proportional to the value of the weight of the third pixel value.

[0104] Optionally, referring to Figure 12 , the target pixel value of the pixel point corresponding to the third pixel value and the fourth pixel value is determined according to the weight of the third pixel value and the weight of the fourth pixel value, comprising: according to the weight of the third pixel value and the weight of the fourth pixel value, the target pixel value Q of the pixel point corresponding to the third pixel value and the fourth pixel value is determined by the following preset formula:

[0105]

[0106] Wherein, A is the first distance, B is the second distance, x is the fourth pixel value, y is the third pixel value, Q is the target pixel value of the pixel point in the second overlap region, is the weight corresponding to the fourth pixel value, is the weight corresponding to the third pixel value.

[0107] Through the method of the embodiment of the application, in the panoramic image of the vehicle, the closer a pixel point is to a road environment image, the greater the weight of the pixel point corresponding to the road environment image is, a corresponding weight is obtained, so that when a corresponding target pixel value is calculated according to the weight and the road environment image is spliced, the smooth transition of the final overlapping area and the road environment image can be ensured, thereby improving the image quality of the panoramic image of the vehicle.

[0108] In order to illustrate the method of the embodiment of the application, the following takes a trailer as an example for illustration:

[0109] 1. Structure arrangement

[0110] 1.1 Camera structure arrangement, see Figure 2 The trailer is composed of a towing vehicle and a carriage, and a fisheye camera is used to ensure that there is sufficient overlapping area for adjacent cameras. In addition, since the central clarity of the fisheye camera is high and the periphery is blurred, in order to ensure the clarity, the peripheral area of the vehicle body falls in the center of the field of view of the camera as much as possible, therefore, six cameras are used, which are respectively installed on the front left and right of the front vehicle and on the rear left and right of the rear vehicle, so as to ensure that there is sufficient overlapping area for each adjacent camera of the trailer and most of the area falls in the center of the field of view of the camera, thereby ensuring the clarity.

[0111] 1.2 Structure arrangement of calibration board, see Figure 2 The angle between the front vehicle and the rear vehicle is calibrated through the calibration board, the images collected by the camera 2 and the camera 4 can see the calibration board 1 and the calibration board 3, the images collected by the camera 3 and the camera 5 can see the calibration board 2 and the calibration board 4, the camera 1 is a camera installed on the head of the towing vehicle, and the camera 6 is a camera installed on the tail of the carriage. Among them, since the outside camera cannot recognize the camera of the other carriage when the trailer turns, it will cause the function to fail, therefore, the calibration board is installed on the left and right sides respectively, different calibration boards are recognized according to different turning conditions of the vehicle, so as to ensure that one side of the left and right sides can complete the calibration in any case, the angle between the front vehicle and the rear vehicle is calculated, and the splicing is ensured to be accurate.

[0112] 2. Extrinsic parameter calibration

[0113] Among them, the trailer splicing is composed of three parts of 270-degree panoramic splicing of the front vehicle, 270-degree panoramic splicing of the rear vehicle and panoramic splicing of the front and rear vehicles. The 270-degree panoramic splicing of the front vehicle and the 270-degree panoramic splicing of the rear vehicle depend on the internal and external parameter data of three cameras, since there is an error in the installation of each camera on the vehicle body, each vehicle needs to be calibrated to accurately know the position of each camera on the vehicle body, i.e. the external parameter, and the mapping relationship from the world coordinate with the center of the vehicle body as the world coordinate center to the image coordinate is obtained according to the internal and external parameters, and the panoramic splicing is completed by point-by-point mapping. See Figure 4The calibration board is placed around the vehicle body, and it is ensured that the images captured by two adjacent cameras contain the unified marker. Through the steps of external parameter calibration, fisheye image acquisition, distortion correction, corner point identification, and external parameter calculation, the relationship conversion of the camera internal and external parameters in each coordinate system is obtained.

[0114] A world coordinate system is established with the center of the vehicle body as the origin, and the position of the camera in the world coordinate system is calculated. Specifically, in the world coordinate system of the marker, after constructing the covariance matrix of the marker, the eigenvector corresponding to the minimum eigenvalue of the covariance matrix and the coordinate mean value of the covariance matrix are calculated; the initial rotation matrix R of the marker is obtained by transformation using the eigenvector and the coordinate mean value; then, the initial translation vector T of the marker is calculated according to the initial rotation matrix R and the coordinate mean value; the initial rotation matrix R and the initial translation vector T are iteratively optimized using the re-projection error between the coordinates of the marker in the calibration image and the coordinates of the marker in the world coordinate system, to obtain the mapping relationship between the coordinates of the marker in the calibration image and the coordinates of the marker in the world coordinate system; and the position relationship between the vehicle-mounted camera and the marker is obtained according to the mapping relationship and the internal parameters of the vehicle-mounted camera.

[0115] 3. Towing vehicle, 270-degree panoramic view generation

[0116] Referring to Figure 5 and Figure 8 , the final panoramic overhead view of the trailer is formed by splicing and fusing the overhead views of the towing vehicle and the trailer. To obtain the final 360-degree panoramic overhead view, the 270-degree panoramic overhead view of the towing vehicle and the 270-degree panoramic overhead view of the trailer are obtained. The 270-degree panoramic overhead view of the towing vehicle is formed by splicing and fusing the front camera, the left front camera, and the right front camera, and the 270-degree panoramic overhead view of the trailer is formed by splicing and fusing the left rear camera, the right rear camera, and the rear camera. Therefore, the sub-overhead view of each camera needs to be calculated first.

[0117] 3.1 Sub-overhead view region splitting, referring to Figure 5 and Figure 8 , the panoramic region is split into six regions according to the camera structure arrangement, and there is enough fusion region between adjacent two cameras. A rectangular region is selected with the center of the vehicle body as the origin and with an outward extension of 3m in front, back, left, and right based on the vehicle length and width. Within this region, the panoramic view is split into a front sub-overhead view, a left front sub-overhead view, a right front sub-overhead view, a left rear sub-overhead view, a right rear sub-overhead view, and a rear sub-overhead view Figure Six .

[0118] 3.2 Sub-overview image calculation, six pairs of sub-overview images are converted from images collected by six cameras installed around the vehicle body. All pixel points in the six pairs of sub-overview images are mapped to find pixel values in the fisheye original image one by one, and the pixel values are projected into six image buffers to obtain six pairs of sub-overview images without distortion.

[0119] 3.3 Sub-overview image fusion, see Figure 5 and Figure 8 The front 270-degree panoramic image and the rear 270-degree panoramic image are respectively rendered by splicing three cameras. In order to avoid seeing obvious seams when splicing adjacent cameras into one image, fusion processing is required. The seam position is selected at the left front and right front 45-degree positions, as shown by the two oblique lines in the figure. Image fusion is performed on both sides of the seam position according to the weight. The fusion weight of the two images is 0-1. The closer to a certain camera, the higher the pixel weight, and vice versa. See Figure 10 The pixel value of the left front sub-overview image is n, the pixel value of the front sub-overview image is m, the angle AOB obtained by connecting the starting point of the seam position and the lower boundary of the fusion area is α, the angle AOC obtained by connecting the starting point of the seam position and the right boundary of the fusion area is β, and the pixel value P of point A in the final panoramic image is

[0120]

[0121] Wherein, α is the first included angle, β is the second included angle, m is the first pixel value, n is the second pixel value, and P is the target pixel value of the pixel point in the first overlapping area, is the weight corresponding to the first pixel value, is the weight corresponding to the second pixel value. For example, see Figure 10 For the pixel point A in the overlapping area, the first target boundary is the OB edge, and the second target boundary is the OA edge. ∠AOB is the first included angle, and ∠AOC is the second included angle.

[0122] 3.4 Towing vehicle and carriage angle calibration calculation. The calibration plate is installed on the edge of the carriage and at the imaging edge of the camera. The steering can signal is read in real time during operation. If it is detected that the turning angle is greater than a certain angle, it is considered that the vehicle is turning. The inside two cameras are selected for angle point recognition of the calibration plate and angle calculation. Through real-time image detection, the position of the calibration plate in the fisheye image is detected to calculate the angle. The calibration plate is installed at a known position of the vehicle body. The camera recognizes the position of the calibration plate through image collection to establish a covariance matrix to iteratively solve a rotation matrix and a translation vector, obtain the position relationship of the camera based on the calibration plate, and convert to obtain the current position of the camera. For example Figure 6As shown, the dashed line represents the position at 0 degrees, and the solid line represents the position of the car head after rotating a certain angle. At 0 degrees, the camera obtains the position based on the center position of the car body through the structure. When turning, the camera position can be obtained by recognizing the calibration board to obtain the relative position based on the calibration board. After transformation, the position based on the world coordinate system of the car body center is obtained. Connecting the rotation points of the front and rear vehicles respectively, the included angle ε is the included angle between the front and rear vehicles.

[0123] 3.5 For the panoramic stitching of the tractor and trailer, since the angle between the front and rear vehicles has been calculated, it is only necessary to rotate the 270-degree panoramic view of the front vehicle to 0 degrees. The above operation can be completed using the planar rotation formula, which is as follows:

[0124] x′=xcosθ+ysinθ

[0125] y′=ycosθ-xsinθ

[0126] Where θ is the rotation angle, (x,y) are the coordinates of the point before rotation, and (x`,y`) are the coordinates of the point after rotation.

[0127] After rotation, a front and rear panoramic view is merged using a gradient blending technique. Within the merging area, the closer to the front panoramic image, the larger the pixel proportion of the front panoramic image, and vice versa. For point O, with distance A from the upper boundary of the merging area and distance B from the lower boundary, and pixel value m in the front panoramic image and pixel value n in the rear panoramic image, the pixel value P of point O in the merged panoramic image is:

[0128]

[0129] Where A is the first distance, B is the second distance, x is the fourth pixel value, y is the third pixel value, and Q is the target pixel value of the pixel in the second overlapping region. The weight corresponding to the fourth pixel value, This is the weight corresponding to the third pixel value.

[0130] A second aspect of the embodiments of this application, see [link to embodiment]. Figure 13 A vehicle panoramic image generation device is provided, comprising:

[0131] Image acquisition module 1301 acquires target images captured by target cameras of target vehicle body units, wherein the target camera refers to the camera located inside the target vehicle body unit when the vehicle is turning.

[0132] Image recognition module 1302 is used to identify the image position information of the target marker in the target image, wherein the target marker and the target camera are located in different vehicle body units;

[0133] The position calculation module 1303 is configured to calculate a first relative position of the target camera and the target calibration object according to image position information of the target calibration object.

[0134] The included angle calculation module 1304 is configured to determine a target included angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle according to the first relative position.

[0135] The image splicing module 1305 is configured to splice the driving environment images collected by the cameras on the first vehicle body unit and the second vehicle body unit to obtain a panoramic image by using the target included angle.

[0136] Optionally, the target vehicle body unit includes the first vehicle body unit and / or the second vehicle body unit in the turning state of the vehicle; and correspondingly, the target image collected by the target camera of the target vehicle body unit includes at least one of the following:

[0137] The target image collected by the target camera of the first vehicle body unit, wherein the target image at least includes a second calibration object on the inner side of the second vehicle body unit in the turning state of the vehicle;

[0138] The target image collected by the target camera of the second vehicle body unit, wherein the target image at least includes a first calibration object on the inner side of the first vehicle body unit in the turning state of the vehicle.

[0139] Optionally, the image recognition module 1302 includes:

[0140] The second calibration object recognition submodule is configured to recognize image position information of the second calibration object on the inner side of the second vehicle body unit in the target image collected by the target camera of the first vehicle body unit;

[0141] The first calibration object recognition submodule is configured to recognize image position information of the first calibration object on the inner side of the first vehicle body unit in the target image collected by the target camera of the second vehicle body unit.

[0142] Optionally, the included angle calculation module 1304 includes:

[0143] The first rotation angle determination submodule is configured to determine a first rotation angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle according to the first relative position of the target camera of the first vehicle body unit and the second calibration object on the inner side of the second vehicle body unit.

[0144] The second rotation angle determination submodule is configured to determine a second rotation angle between the first vehicle body unit and the second vehicle body unit in the turning state of the vehicle according to the first relative position of the target camera of the second vehicle body unit and the first calibration object on the inner side of the first vehicle body unit.

[0145] The rotation angle average determination submodule is configured to determine an average of the first rotation angle and the second rotation angle as a target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in the turning state.

[0146] Optionally, the included angle calculation module 1304 comprises:

[0147] The coordinate position calculation submodule is configured to calculate, according to the first relative position, a second relative position of the target camera and the target calibration object when the vehicle is in the non-turning state, and a third relative position of the target calibration object and the preset rotation point when the vehicle is in the non-turning state, a first coordinate position of the preset rotation point, a second coordinate position corresponding to the target camera when the vehicle is in the non-turning state, and a third coordinate position corresponding to the target camera when the vehicle is in the turning state in the world coordinate system; wherein the preset rotation point is a rotation center of the first vehicle body unit and the second vehicle body unit.

[0148] The target included angle calculation submodule is configured to calculate an included angle between a line connecting the first coordinate position and the second coordinate position and a line connecting the first coordinate position and the third coordinate position, to obtain the target included angle.

[0149] Optionally, the position calculation module 1303 comprises:

[0150] The first predicted coordinate position calculation submodule is configured to calculate a first predicted coordinate position of the second calibration object in the first camera coordinate system according to the image position information of the second calibration object.

[0151] The first covariance matrix determination submodule is configured to create a first covariance matrix according to the first predicted coordinate position, and to determine a feature vector corresponding to a minimum eigenvalue of the first covariance matrix and a coordinate mean value corresponding to the first covariance matrix.

[0152] The first coordinate mapping relationship calculation submodule is configured to calculate a first coordinate mapping relationship between the image coordinate system and the first camera coordinate system of the second calibration object according to the feature vector corresponding to the minimum eigenvalue of the first covariance matrix and the coordinate mean value corresponding to the first covariance matrix.

[0153] The first relative position calculation submodule is configured to calculate the first relative position of the second calibration object on the inner side of the second vehicle body unit according to the image position information of the second calibration object and the first coordinate mapping relationship.

[0154] And / or,

[0155] The second predicted coordinate position calculation submodule is configured to calculate a second predicted coordinate position of the first calibration object in the second camera coordinate system according to the image position information of the first calibration object.

[0156] The minimum eigenvalue determination submodule is configured to create a second covariance matrix according to the second predicted coordinate position, and determine a feature vector corresponding to a minimum eigenvalue of the second covariance matrix and a coordinate mean value corresponding to the second covariance matrix;

[0157] The second coordinate mapping relationship determination submodule is configured to calculate a second coordinate mapping relationship between the first calibration object in the image coordinate system and the second camera coordinate system according to the feature vector corresponding to the minimum eigenvalue of the second covariance matrix and the coordinate mean value corresponding to the second covariance matrix.

[0158] The second coordinate mapping relationship calculation submodule is configured to calculate a first relative position of the first calibration object inside the first vehicle body unit from the target camera of the second vehicle body unit according to the image position information of the first calibration object and the second coordinate mapping relationship.

[0159] Optionally, the device further comprises:

[0160] The first overlap region identification module is configured to identify a first overlap region of images collected by cameras on the target vehicle body unit; if the target vehicle body unit is the first vehicle body unit and the first vehicle body unit is a vehicle head body unit, the first overlap region includes an overlap region of a third image collected by a camera at the vehicle head of the first vehicle body unit and a fourth image collected by a camera on the first side surface, and / or an overlap region of the third image collected by the camera at the vehicle head of the first vehicle body unit and a fourth image collected by a camera on the second side surface; or if the target vehicle body unit is the second vehicle body unit and the second vehicle body unit is a vehicle tail body unit, the first overlap region includes an overlap region of a third image collected by a camera at the vehicle tail of the second vehicle body unit and a fourth image collected by a camera on the first side surface, and / or an overlap region of the third image collected by the camera at the vehicle tail of the second vehicle body unit and a fourth image collected by a camera on the second side surface.

[0161] The second pixel value weight calculation module is configured to calculate, for any pixel point in the first overlap region of the third image and the fourth image with the overlap region, a weight of a first pixel value corresponding to the pixel point in the third image and a weight of a second pixel value corresponding to the pixel point in the fourth image according to a position of the pixel point in the first overlap region; if the position of the pixel point in the first overlap region is closer to the third image, the weight of the first pixel value corresponding to the pixel point in the third image is greater; if the position of the pixel point in the first overlap region is closer to the fourth image, the weight of the second pixel value corresponding to the pixel point in the fourth image is greater.

[0162] The target pixel value determination module is configured to determine a target pixel value of a pixel point corresponding to the first pixel value and the second pixel value according to the weight of the first pixel value and the weight of the second pixel value.

[0163] The driving environment image acquisition module is configured to stitch the third image and the fourth image according to the target pixel values of the pixels in the first overlap region to obtain a driving environment image corresponding to the target vehicle body unit.

[0164] Optionally, the first overlap region includes a first target boundary and a second target boundary, and the first target boundary and the second target boundary intersect; and the first target boundary and the second target boundary include at least one of the following cases:

[0165] (a) In a case where the first overlap region includes an overlap region of the third image collected by the camera at the front of the first vehicle body unit and the fourth image collected by the camera on the first side:

[0166] If the first target boundary is a lower boundary of the first overlap region in the image height direction, the second target boundary is a right boundary of the first overlap region in the image width direction; or

[0167] If the first target boundary is a left boundary of the first overlap region in the image width direction, the second target boundary is an upper boundary of the first overlap region in the image height direction.

[0168] (b) In a case where the first overlap region includes an overlap region of the third image collected by the camera at the front of the first vehicle body unit and the fourth image collected by the camera on the second side:

[0169] If the first target boundary is a lower boundary of the first overlap region in the image height direction, the second target boundary is a left boundary of the first overlap region in the image width direction; or

[0170] If the first target boundary is a right boundary of the first overlap region in the image width direction, the second target boundary is an upper boundary of the first overlap region in the image height direction.

[0171] (c) In a case where the first overlap region includes an overlap region of the third image collected by the camera at the rear of the second vehicle body unit and the fourth image collected by the camera on the first side:

[0172] If the first target boundary is an upper boundary of the first overlap region in the image height direction, the second target boundary is a right boundary of the first overlap region in the image width direction; or

[0173] If the first target boundary is a left boundary of the first overlap region in the image width direction, the second target boundary is a lower boundary of the first overlap region in the image height direction.

[0174] (d) in the case that the first overlap region comprises an overlap region of a third image captured by a camera at a rear of the second vehicle body unit and a fourth image captured by a camera of the second side face:

[0175] If the first target boundary is an upper boundary of the first overlap region in the image height direction, the second target boundary is a left boundary of the first overlap region in the image width direction; or

[0176] If the first target boundary is a right boundary of the first overlap region in the image width direction, the second target boundary is a lower boundary of the first overlap region in the image height direction.

[0177] The image height direction is perpendicular to the image width direction. For any pixel point in the first overlap region of the third image and the fourth image, a weight of a first pixel value corresponding to the pixel point in the third image and a weight of a second pixel value corresponding to the pixel point in the fourth image are calculated according to a position of the pixel point in the first overlap region, comprising:

[0178] A coordinate of an intersection of the first target boundary and the second target boundary is identified to obtain a target origin.

[0179] For any pixel point in the first overlap region, a first included angle between a line connecting the pixel point and the target origin and the first target boundary is calculated, and a second included angle between the line connecting the pixel point and the target origin and the second target boundary is calculated.

[0180] The weight of the first pixel value corresponding to the pixel point in the third image is determined according to the first included angle, and the weight of the second pixel value corresponding to the pixel point in the fourth image is determined according to the second included angle.

[0181] Optionally, the target pixel value determination module is specifically configured to determine a target pixel value P of the pixel point corresponding to the first pixel value and the second pixel value according to the weight of the first pixel value and the weight of the second pixel value through the following preset formula:

[0182]

[0183] wherein, a is the first included angle, β is the second included angle, m is the first pixel value, n is the second pixel value, P is the target pixel value of the pixel point in the first overlap region, is the weight corresponding to the first pixel value, is the weight corresponding to the second pixel value.

[0184] Optionally, the image splicing module 1305 comprises:

[0185] The translation transformation submodule is configured to perform translation transformation on the driving environment images captured by the cameras on the first vehicle body unit and / or the second vehicle body unit according to the target included angle.

[0186] The driving environment image recognition submodule is configured to recognize the second overlap region in the driving environment images of the first vehicle body unit and the second vehicle body unit after the translation transformation operation.

[0187] The weight calculation submodule is configured to calculate, for any pixel point in the second overlap region, a weight of a third pixel value corresponding to the pixel point in the driving environment image of the first vehicle body unit after the translation transformation operation and a weight of a fourth pixel value corresponding to the pixel point in the driving environment image of the second vehicle body unit after the translation transformation operation according to a position of the pixel point in the second overlap region, wherein the weight of the third pixel value corresponding to the pixel point is greater if the position of the pixel point in the second overlap region is closer to the driving environment image of the first vehicle body unit, and the weight of the fourth pixel value corresponding to the pixel point is greater if the position of the pixel point in the second overlap region is closer to the driving environment image of the second vehicle body unit.

[0188] The pixel point determination submodule is configured to determine a target pixel value of the pixel point corresponding to the third pixel value and the fourth pixel value according to the weight of the third pixel value and the weight of the fourth pixel value.

[0189] The image splicing submodule is configured to splice the driving environment images of the first vehicle body unit and the second vehicle body unit after the translation transformation operation according to the target pixel values of the pixel points in the second overlap region calculated to obtain a panoramic image of the vehicle.

[0190] Optionally, the second overlap region includes a third target boundary and a fourth target boundary, the third target boundary is an upper boundary of the second overlap region in an image height direction, the fourth target boundary is a lower boundary of the second overlap region in the image height direction, and the image height direction is perpendicular to an image width direction.

[0191] The first vehicle body unit is a vehicle head unit, and the second vehicle body unit is a vehicle tail unit, the upper boundary of the driving environment image of the first vehicle body unit in the image height direction is higher than the third target boundary, and the lower boundary of the driving environment image of the second vehicle body unit in the image height direction is lower than the fourth target boundary.

[0192] The weight calculation submodule includes:

[0193] The second distance determination unit is configured to recognize, for any pixel point in the second overlap region, a first distance of the pixel point from the third target boundary and a second distance of the pixel point from the fourth target boundary.

[0194] determine a weight of a fourth pixel value in the second vehicle body unit's driving environment image after the translation transformation operation of the pixel point, and a weight of a third pixel value in the first vehicle body unit's driving environment image after the translation transformation operation of the pixel point according to the first distance and the second distance, wherein the value of the first distance is directly proportional to the value of the weight of the fourth pixel value, and the value of the second distance is directly proportional to the value of the weight of the third pixel value.

[0195] Optionally, the pixel point determination sub-module is specifically configured to determine a target pixel value Q of a pixel point corresponding to the third pixel value and the fourth pixel value according to the weight of the third pixel value and the weight of the fourth pixel value through the following preset formula:

[0196]

[0197] wherein A is the first distance, B is the second distance, x is the fourth pixel value, y is the third pixel value, and Q is the target pixel value of the pixel point in the second overlapping area, is the weight corresponding to the fourth pixel value, is the weight corresponding to the third pixel value.

[0198] It can be seen that, by means of the device of the embodiments of the present application, the image position information of the target calibration object in the target image can be recognized, so as to calculate the first relative position of the target camera and the target calibration object according to the recognized image position information, and determine the target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in the turning state, and use the target included angle to splice the driving environment images collected by the cameras on the first vehicle body unit and the second vehicle body unit to obtain a panoramic image, thereby realizing the generation of a panoramic image for the vehicle whose vehicle body is not on the same straight line when turning.

[0199] The embodiments of the present application also provide an electronic device, as shown in the accompanying drawings, Figure 14 The processor 1401, the communication interface 1402, and the memory 1403 can communicate with each other through the communication bus 1404,

[0200] The memory 1403 is configured to store a computer program;

[0201] The processor 1401 is configured to execute the program stored in the memory 1403 to implement the following steps:

[0202] obtain a target image collected by a target camera of a target vehicle body unit, wherein the target camera refers to a camera on the inner side of the target vehicle body unit when the vehicle is in a turning state;

[0203] identify image position information of a target marker in the target image, wherein the target marker and the target camera are located in different body units;

[0204] calculate a first relative position of the target camera and the target marker according to the image position information of the target marker;

[0205] determine a target included angle between the first body unit and the second body unit when the vehicle is in a turning state according to the first relative position;

[0206] stitch the driving environment images captured by the cameras on the first body unit and the second body unit to obtain a panoramic image using the target included angle.

[0207] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0208] The communication interface is used for communication between the above electronic device and other devices.

[0209] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0210] The above processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0211] In a further embodiment provided in the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any of the vehicle panoramic image generation methods described above.

[0212] In a further embodiment provided in the present application, a computer program product containing instructions, which, when executed on a computer, cause the computer to perform any of the vehicle panoramic image generation methods described in the above embodiments.

[0213] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0214] It should be noted that in this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0215] Each of the embodiments in the specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device, electronic device, storage medium and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0216] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating a vehicle panoramic image, the method comprising: The application is applied to a vehicle comprising a first vehicle body unit and a second vehicle body unit, at least a first side and a second side of each vehicle body unit are respectively provided with a camera and a calibration object, the first side and the second side are left and right sides of the vehicle body along the driving direction of the vehicle; The method comprises: acquiring a target image collected by a target camera of a target vehicle body unit, wherein the target vehicle body unit comprises the first vehicle body unit and / or the second vehicle body unit when the vehicle is in a turning state, and the target camera refers to a camera on the inner side of the target vehicle body unit when the vehicle is in the turning state; identifying image position information of a target calibration object in the target image, wherein the target calibration object and the target camera are located in different vehicle body units; calculating a first relative position of the target camera and the target calibration object according to the image position information of the target calibration object; the first relative position comprises a first relative position of a target camera of the first vehicle body unit and a second calibration object on the inner side of the second vehicle body unit, and / or a first relative position of a target camera of the second vehicle body unit and a first calibration object on the inner side of the first vehicle body unit; determining a target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in the turning state according to the first relative position; splicing driving environment images collected by cameras on the first vehicle body unit and the second vehicle body unit to obtain a panoramic image by using the target included angle; wherein the determination of the target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in the turning state according to the first relative position comprises: determining a first rotation angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in the turning state according to the first relative position of the target camera of the first vehicle body unit and the second calibration object on the inner side of the second vehicle body unit; determining a second rotation angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in the turning state according to the first relative position of the target camera of the second vehicle body unit and the first calibration object on the inner side of the first vehicle body unit; and determining the average value of the first rotation angle and the second rotation angle as the target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in the turning state. Alternatively, according to the first relative position, a second relative position of the target camera and the target calibration object in a non-turning state of the vehicle, and a third relative position of the target calibration object and a preset rotation point in the non-turning state of the vehicle, a first coordinate position of the preset rotation point, a second coordinate position corresponding to the target camera in the non-turning state of the vehicle, and a third coordinate position corresponding to the target camera in a turning state of the vehicle are calculated in a preset coordinate system, wherein the preset rotation point is a rotation center of the first vehicle body unit and the second vehicle body unit; an included angle between a line connecting the first coordinate position and the second coordinate position and a line connecting the first coordinate position and the third coordinate position is calculated to obtain the target included angle; The method further comprises: identifying a first overlapping area of images collected by each camera on the target vehicle body unit; if the target vehicle body unit is a first vehicle body unit, and the first vehicle body unit is a vehicle head unit, the first overlapping area includes an overlapping area of a third image collected by a camera at the vehicle head of the first vehicle body unit and a fourth image collected by a camera on a first side surface, and / or an overlapping area of the third image collected by the camera at the vehicle head of the first vehicle body unit and a fourth image collected by a camera on a second side surface; or if the target vehicle body unit is a second vehicle body unit, and the second vehicle body unit is a vehicle tail unit, the first overlapping area includes an overlapping area of a third image collected by a camera at the vehicle tail of the second vehicle body unit and a fourth image collected by a camera on a first side surface, and / or an overlapping area of the third image collected by the camera at the vehicle tail of the second vehicle body unit and a fourth image collected by a camera on a second side surface; for any pixel point in the first overlapping area of the third image and the fourth image, according to a position of the pixel point in the first overlapping area, a weight of a first pixel value corresponding to the pixel point in the third image and a weight of a second pixel value corresponding to the pixel point in the fourth image are calculated; if the position of the pixel point in the first overlapping area is closer to the third image, the weight of the first pixel value corresponding to the pixel point in the third image is greater; if the position of the pixel point in the first overlapping area is closer to the fourth image, the weight of the second pixel value corresponding to the pixel point in the fourth image is greater; according to the weight of the first pixel value and the weight of the second pixel value, a target pixel value of the pixel point corresponding to the first pixel value and the second pixel value is determined; according to the target pixel value of each pixel point in the first overlapping area calculated, the third image and the fourth image are spliced to obtain a driving environment image corresponding to the target vehicle body unit.

2. The method of claim 1, wherein: The target image collected by the target camera of the target vehicle body unit comprises at least one of the following: acquire a target image captured by a target camera of the first vehicle body unit; wherein the target image at least includes a second calibration object on an inner side of the second vehicle body unit when the vehicle is in a turning state; acquire a target image captured by a target camera of the second vehicle body unit; wherein the target image at least includes a first calibration object on an inner side of the first vehicle body unit when the vehicle is in a turning state.

3. The method of claim 2, wherein, the identifying the image position information of the target calibration object in the target image comprises: identifying the image position information of the second calibration object on the inner side of the second vehicle body unit in the target image captured by the target camera of the first vehicle body unit; and / or, identifying the image position information of the first calibration object on the inner side of the first vehicle body unit in the target image captured by the target camera of the second vehicle body unit.

4. The method of claim 3, wherein, the calculating the first relative position of the target camera and the target calibration object according to the image position information of the target calibration object comprises: calculating a first predicted coordinate position of the second calibration object in a first camera coordinate system according to the image position information of the second calibration object; creating a first covariance matrix according to the first predicted coordinate position, and determining a feature vector corresponding to a minimum eigenvalue of the first covariance matrix and a coordinate mean value corresponding to the first covariance matrix; calculating a first coordinate mapping relationship between the image coordinate system and the first camera coordinate system according to the feature vector corresponding to the minimum eigenvalue of the first covariance matrix and the coordinate mean value corresponding to the first covariance matrix; calculating the first relative position of the target camera of the first vehicle body unit and the second calibration object on the inner side of the second vehicle body unit according to the image position information of the second calibration object and the first coordinate mapping relationship; and / or, calculating a second predicted coordinate position of the first calibration object in a second camera coordinate system according to the image position information of the first calibration object; creating a second covariance matrix according to the second predicted coordinate position, and determining a feature vector corresponding to a minimum eigenvalue of the second covariance matrix and a coordinate mean value corresponding to the second covariance matrix; calculating a second coordinate mapping relationship between the image coordinate system and the second camera coordinate system according to the feature vector corresponding to the minimum eigenvalue of the second covariance matrix and the coordinate mean value corresponding to the second covariance matrix; calculating the first relative position of the target camera of the second vehicle body unit and the first calibration object on the inner side of the first vehicle body unit according to the image position information of the first calibration object and the second coordinate mapping relationship.

5. The method of claim 1, wherein, the first overlapping region includes a first target boundary and a second target boundary, and the first target boundary and the second target boundary intersect; the first target boundary and the second target boundary include at least one of the following cases: (a) in the case where the first overlapping region includes an overlapping region of a third image captured by a camera on a vehicle head of the first vehicle body unit and a fourth image captured by a camera on a first side surface: If the first target boundary is a lower boundary of the first overlap region in the image height direction, the second target boundary is a left boundary of the first overlap region in the image width direction; or If the first target boundary is a left boundary of the first overlap region in the image width direction, the second target boundary is a lower boundary of the first overlap region in the image height direction. (b) In the case where the first overlap region comprises an overlap region of a third image captured by a camera at a front of the first vehicle body unit and a fourth image captured by a camera at a first side: If the first target boundary is a lower boundary of the first overlap region in the image height direction, the second target boundary is a left boundary of the first overlap region in the image width direction; or If the first target boundary is a right boundary of the first overlap region in the image width direction, the second target boundary is an upper boundary of the first overlap region in the image height direction. (c) In the case where the first overlap region comprises an overlap region of a third image captured by a camera at a rear of the second vehicle body unit and a fourth image captured by a camera at a first side: If the first target boundary is an upper boundary of the first overlap region in the image height direction, the second target boundary is a right boundary of the first overlap region in the image width direction; or If the first target boundary is a left boundary of the first overlap region in the image width direction, the second target boundary is a lower boundary of the first overlap region in the image height direction. (d) In the case where the first overlap region comprises an overlap region of a third image captured by a camera at a rear of the second vehicle body unit and a fourth image captured by a camera at a second side: If the first target boundary is an upper boundary of the first overlap region in the image height direction, the second target boundary is a left boundary of the first overlap region in the image width direction; or If the first target boundary is a right boundary of the first overlap region in the image width direction, the second target boundary is a lower boundary of the first overlap region in the image height direction. The image height direction is a direction perpendicular to the image width direction, and the calculation of the weight of the first pixel value corresponding to any pixel point in the first overlap region of the third image and the weight of the second pixel value corresponding to the pixel point in the fourth image according to the position of the pixel point in the first overlap region comprises: identifying the coordinates of the intersection of the first target boundary and the second target boundary to obtain a target origin; for any pixel point in the first overlap region, calculating a first included angle between a line connecting the pixel point and the target origin and the first target boundary, and a second included angle between the line connecting the pixel point and the target origin and the second target boundary according to the position of the pixel point in the first overlap region; and ​ ​ According to the first included angle, a weight of a first pixel value corresponding to the pixel point in a third image is determined, and according to the second included angle, a weight of a second pixel value corresponding to the pixel point in a fourth image is determined.

6. The method of claim 5, wherein, According to the first pixel value weight and the second pixel value weight, the target pixel value of the pixel point corresponding to the first pixel value and the second pixel value is determined. According to the first pixel value weight and the second pixel value weight, the target pixel value of the pixel point corresponding to the first pixel value and the second pixel value is determined. According to the first pixel value weight and the second pixel value weight, the target pixel value of the pixel point corresponding to the first pixel value and the second pixel value is determined. wherein a is the first included angle, β is the second included angle, m is the first pixel value, n is the second pixel value, P is a target pixel value of a pixel point in the first overlapping region, is a weight corresponding to the first pixel value, is a weight corresponding to the second pixel value.

7. The method of claim 1, wherein, According to the target included angle, the driving environment images collected by the cameras on the first vehicle body unit and the second vehicle body unit are spliced to obtain a panoramic image, comprising: According to the target included angle, the driving environment images collected by the cameras on the first vehicle body unit and / or the second vehicle body unit are translated and transformed; A second overlapping area in the driving environment images of the corresponding first vehicle body unit and second vehicle body unit after the translation and transformation operation is identified; For any pixel point in the second overlapping area, according to the position of the pixel point in the second overlapping area, the weight of a third pixel value corresponding to the pixel point in the driving environment image of the first vehicle body unit after the translation and transformation operation is calculated, and the weight of a fourth pixel value corresponding to the pixel point in the driving environment image of the second vehicle body unit after the translation and transformation operation is calculated; Wherein, if the position of the pixel point of the second overlapping area is closer to the driving environment image of the first vehicle body unit, the weight of the third pixel value corresponding to the pixel point is greater, and if the position of the pixel point of the second overlapping area is closer to the driving environment image of the second vehicle body unit, the weight of the fourth pixel value corresponding to the pixel point is greater; According to the third pixel value weight and the fourth pixel value weight, the target pixel value of the pixel point corresponding to the third pixel value and the fourth pixel value is determined. According to the target pixel values of the pixel points in the second overlapping area calculated, the driving environment images of the first vehicle body unit and the second vehicle body unit after the translation and transformation operation are spliced to obtain a panoramic image of the vehicle.

8. The method of claim 7, wherein, The second overlapping area includes a third target boundary and a fourth target boundary, the third target boundary is the upper boundary of the second overlapping area in the image height direction, and the fourth target boundary is the lower boundary of the second overlapping area in the image height direction, and the image height direction is perpendicular to the image width direction; The first vehicle body unit is a vehicle head body unit, and the second vehicle body unit is a vehicle tail body unit, the upper boundary of the driving environment image of the first vehicle body unit in the image height direction is higher than the third target boundary, and the lower boundary of the driving environment image of the second vehicle body unit in the image height direction is lower than the fourth target boundary; The weight of the third pixel value in the driving environment image of the first vehicle body unit after the translation transformation operation and the weight of the fourth pixel value in the driving environment image of the second vehicle body unit after the translation transformation operation are calculated according to the position of the pixel point in the second overlapping area. For any pixel point in the second overlapping area, the first distance of the pixel point from the third target boundary and the second distance of the pixel point from the fourth target boundary are identified. According to the first distance and the second distance, the weight of the fourth pixel value in the driving environment image of the second vehicle body unit after the translation transformation operation and the weight of the third pixel value in the driving environment image of the first vehicle body unit after the translation transformation operation are determined, wherein the value of the first distance is directly proportional to the value of the weight of the fourth pixel value, and the value of the second distance is directly proportional to the value of the weight of the third pixel value.

9. The method of claim 8, wherein, The target pixel value of the pixel point corresponding to the third pixel value and the fourth pixel value is determined according to the weight of the third pixel value and the weight of the fourth pixel value. The target pixel value Q of the pixel point corresponding to the third pixel value and the fourth pixel value is determined according to the weight of the third pixel value and the weight of the fourth pixel value through the following preset formula: wherein A is the first distance, B is the second distance, x is the fourth pixel value, y is the third pixel value, Q is a target pixel value of a pixel point in the second overlapping area, is a weight corresponding to the fourth pixel value, is a weight corresponding to the third pixel value.

10. A vehicle surround view image generation apparatus, comprising: The application is applied to a vehicle including a first vehicle body unit and a second vehicle body unit, and a camera and a calibration object are respectively arranged on at least a first side and a second side of each vehicle body unit, wherein the first side and the second side are left and right vehicle body sides along the driving direction of the vehicle. The device comprises: An image acquisition module acquires a target image collected by a target camera of a target vehicle body unit, wherein the target vehicle body unit includes the first vehicle body unit and / or the second vehicle body unit when the vehicle is in a turning state, and the target camera refers to a camera located on the inner side of the target vehicle body unit when the vehicle is in a turning state. An image recognition module is configured to identify image position information of a target calibration object in the target image, wherein the target calibration object and the target camera are located in different vehicle body units. A position calculation module is configured to calculate a first relative position between the target camera and the target calibration object according to the image position information of the target calibration object, wherein the first relative position includes a first relative position between the target camera of the first vehicle body unit and a second calibration object on the inner side of the second vehicle body unit, and / or a first relative position between the target camera of the second vehicle body unit and a first calibration object on the inner side of the first vehicle body unit. An included angle calculation module is configured to determine a target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in a turning state according to the first relative position. An image stitching module is configured to stitch driving environment images collected by cameras on the first vehicle body unit and the second vehicle body unit to obtain a panoramic image by using the target included angle. The included angle calculation module is specifically configured to determine a first rotation angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in a turning state according to a first relative position of a target camera of the first vehicle body unit and a second calibration object on an inner side of the second vehicle body unit; determine a second rotation angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in a turning state according to a first relative position of a target camera of the second vehicle body unit and a first calibration object on an inner side of the first vehicle body unit; and determine an average value of the first rotation angle and the second rotation angle as a target included angle between the first vehicle body unit and the second vehicle body unit when the vehicle is in a turning state. Alternatively, a first coordinate position of a preset rotation point in a preset coordinate system, a second coordinate position of the target camera corresponding to a non-turning state of the vehicle, and a third coordinate position of the target camera corresponding to a turning state of the vehicle are calculated according to the first relative position, a second relative position of the target camera and the target calibration object in a non-turning state of the vehicle, and a third relative position of the target calibration object and the preset rotation point in a non-turning state of the vehicle; wherein the preset rotation point is a rotation center of the first vehicle body unit and the second vehicle body unit; and an included angle between a line connecting the first coordinate position and the second coordinate position and a line connecting the first coordinate position and the third coordinate position is calculated to obtain the target included angle. The device further comprises: The first overlap region identification module is configured to identify a first overlap region of images captured by cameras on a target vehicle body unit; wherein, if the target vehicle body unit is a first vehicle body unit and the first vehicle body unit is a vehicle head body unit, the first overlap region includes an overlap region of a third image captured by a camera at the vehicle head of the first vehicle body unit and a fourth image captured by a camera on a first side, and / or an overlap region of the third image captured by the camera at the vehicle head of the first vehicle body unit and a fourth image captured by a camera on a second side; or, if the target vehicle body unit is a second vehicle body unit and the second vehicle body unit is a vehicle tail body unit, the first overlap region includes an overlap region of a third image captured by a camera at the vehicle tail of the second vehicle body unit and a fourth image captured by a camera on a first side, and / or an overlap region of the third image captured by the camera at the vehicle tail of the second vehicle body unit and a fourth image captured by a camera on a second side. The weight calculation module of the second pixel value is configured to calculate, for any pixel point in the first overlapping region of the third image and the fourth image, a weight of a first pixel value corresponding to the pixel point in the third image and a weight of a second pixel value corresponding to the pixel point in the fourth image according to a position of the pixel point in the first overlapping region; wherein the weight of the first pixel value corresponding to the pixel point in the third image is greater if the position of the pixel point in the first overlapping region is closer to the third image, and the weight of the second pixel value corresponding to the pixel point in the fourth image is greater if the position of the pixel point in the first overlapping region is closer to the fourth image; The target pixel value determination module is configured to determine a target pixel value of the pixel point corresponding to the first pixel value and the second pixel value according to the weight of the first pixel value and the weight of the second pixel value. The driving environment image acquisition module is configured to splice the third image and the fourth image according to the target pixel values of the pixel points in the first overlapping region to obtain the driving environment image corresponding to the target vehicle body unit.

11. An electronic device, comprising: The system includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory to implement the method steps of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the method steps of any one of claims 1-9.

Images