End-to-end panoramic image stitching method, electronic device, and storage medium

Abstract

The present invention provides an end-to-end panoramic image stitching method, an electronic device, and a storage medium. The panoramic image stitching method comprises the following steps: acquiring pictures captured by multiple cameras of a vehicle; processing the multiple pictures to obtain camera data; and inputting the camera data, preset parameter data and weight data into a trained panoramic image stitching model to generate a panoramic image, wherein the parameter data is a coordinate mapping relationship between the pictures and the panoramic image, and the weight data is weights of the pictures in the panoramic image. The method can obtain a panoramic stitched view in an end-to-end manner by directly inputting camera data, parameter data and weight data into a stitching model, without intermediate transformations and brute-force matching. The method involves few model parameters, requires less inference time and less GPU computing power, achieves relatively high real-time performance, and is suitable for embedded platforms.

Description

An end-to-end panoramic image stitching method, electronic device and storage medium TECHNICAL FIELD

[0001] The present application relates to the field of panoramic image stitching technology for automobiles, and in particular to an end-to-end panoramic image stitching method, electronic device and storage medium. BACKGROUND

[0002] In the field of autonomous driving, panoramic image technology uses multiple camera sensors on a vehicle to collect image information around the vehicle, and then combines these image information through image stitching technology to form a 360-degree panoramic image. The main difficulties of this technology are: (1) due to the fisheye camera, the stitched image has distortion and stretching problems; (2) due to the lack of depth information, the stitching area will have misalignment, ghosting and other problems.

[0003] Chinese patent "A panoramic stitching method based on depth estimation" (publication number: CN115861063A) proposes a panoramic stitching method based on depth estimation, which mainly estimates the depth of the bird's eye view of each camera, and then matches the overlapping area. This method can to some extent solve the problem of distortion in the overlapping area of adjacent cameras and poor stitching effect. However, in actual scenarios, the intrinsic parameters of fisheye cameras are not completely accurate, the picture itself cannot completely eliminate the distortion of the bird's eye view, and there will be slight deviations in feature extraction of the bird's eye view, which will affect the network robustness. At the same time, neural network inference is required for each pair of pictures, which consumes more computing power on the autonomous driving domain controller, and finally a brute force matching is required to obtain accurate matching points, which reduces real-time performance.

[0004] Chinese patent "Panoramic stitching method based on vehicle vision sensor and related equipment" (publication number: CN115731104A) proposes a panoramic stitching method based on vehicle vision sensor and related equipment, which mainly outputs multiple overhead feature maps through a deep learning feature model, and stitches multiple overhead images into a panoramic image. The advantage of this method is that deep learning has strong feature extraction capability and is suitable for more scenarios; this method requires a CNN convolutional neural network and a Transformer self-attention model, which has complex network and too many parameters, requiring more computing resources and increasing performance time consumption.

[0005] In summary, existing panoramic image stitching technologies introduce deep learning to solve the misalignment and ghosting problems in the overlapping area. However, these technologies not only have complex algorithms, but also require additional conversion processes, resulting in a cumbersome and redundant overall process, and involve more computing resources with low real-time performance. SUMMARY

[0006] In view of the defects of the prior art, the present application provides an end-to-end panoramic image stitching method, an electronic device and a storage medium to solve the technical problems of complex network structure, excessive computing power consumption and low real-time performance.

[0007] To achieve the above object and other related objects, the present application provides an end-to-end panoramic image stitching method, comprising the following steps: obtaining pictures taken by multiple cameras of a vehicle; processing multiple pictures to obtain camera data; inputting the camera data, preset parameter data and weight data into a trained panoramic image stitching model to generate a panoramic image, wherein the parameter data is the coordinate mapping relationship between the pictures and the panoramic image, and the weight data is the weight of the pictures in the panoramic image.

[0008] In an embodiment of the present application, the preset parameter data is obtained by the following steps: obtaining the inverse mapping relationship from world coordinates to camera coordinates according to the intrinsic and extrinsic parameters of the multiple cameras; calculating the correspondence between the world coordinates and the pixel positions of the pictures taken by the multiple cameras according to the inverse mapping relationship; and obtaining the parameter data according to the correspondence.

[0009] In an embodiment of the present application, the step of calculating the correspondence between the world coordinates and the pixel positions of the pictures taken by the multiple cameras according to the inverse mapping relationship comprises: for each world coordinate within the camera shooting range, calculating the first correspondence between the world coordinate and the pixel position of the picture taken by the camera according to the inverse mapping relationship; for the world coordinates beyond the camera shooting range, the corresponding pixel positions are all marked as (0, 0) to obtain the second correspondence between the world coordinates and the pixel positions of the pictures taken by the multiple cameras; and combining the first correspondence and the second correspondence to obtain a mapping table matrix to represent the correspondence between the world coordinates and the pixel positions of the pictures taken by the multiple cameras; wherein the mapping table matrix corresponds to the camera.

[0010] In an embodiment of the present application, the step of obtaining the parameter data according to the correspondence comprises: combining multiple mapping table matrices and normalizing to obtain the parameter data.

[0011] In an embodiment of the present application, the preset weight data is obtained by the following steps: dividing the panoramic image to be generated into multiple regions according to the positional relationship of the multiple cameras; and obtaining the weight data according to the relationship between the regions and the camera shooting ranges.

[0012] In an embodiment of the present application, the step of dividing the panoramic image to be generated into multiple regions according to the positional relationship of the multiple cameras comprises: determining an image size of the panoramic image to be generated according to the positional relationship of the multiple cameras and the vehicle; and dividing the panoramic image to be generated into multiple regions according to the positional relationship of the multiple cameras and the vehicle according to the image size.

[0013] In an embodiment of the present application, the step of dividing the panoramic image to be generated into multiple regions according to the positional relationship of the multiple cameras according to the image size comprises: setting a region corresponding to the vehicle in the panoramic image to be generated as a vehicle region according to the image size; setting a region covered only by the shooting range of one camera in the panoramic image to be generated as a non-overlapping region according to the shooting range of the multiple cameras; and setting a region covered by the shooting range of at least two cameras in the panoramic image to be generated as an overlapping region according to the shooting range of the multiple cameras.

[0014] In an embodiment of the present application, the step of obtaining the weight data according to the relationship between the region and the shooting range of the camera comprises: when the region is a vehicle region, the weight data of all pixel points in the vehicle region is 0; when the region is a non-overlapping region, the weight data of all pixel points in the non-overlapping region is related to the camera corresponding to the region and has the same value; and when the region is an overlapping region, the weight data of all pixel points in the overlapping region is related to the specific position of the pixel point and the contribution degree of the multiple cameras corresponding to the region.

[0015] To achieve the above object and other related objects, the present application further provides an electronic device, comprising a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to realize the method provided in any one of the above embodiments.

[0016] To achieve the above object and other related objects, the present application further provides a computer readable storage medium, which stores a computer program used to make a computer execute the method provided in any one of the above embodiments.

[0017] Beneficial effects of the present application: the end-to-end panoramic image stitching method, electronic device and storage medium provided by the present application adopt an end-to-end mode, do not need intermediate conversion, do not need brute force matching, directly input camera data, parameter data and weight data into a stitching model, and a panoramic stitching view can be obtained; the method has few model parameters, short inference time, does not need very much GPU computing power, has high real-time performance, and is very suitable for embedded platforms. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0019] Fig. 1 is a flowchart provided by an embodiment of the present application;

[0020] Fig. 2 is a camera data acquisition flowchart provided by an embodiment of the present application;

[0021] Fig. 3 is a schematic diagram of camera data provided by an embodiment of the present application;

[0022] Fig. 4 is a parameter data acquisition flowchart provided by an embodiment of the present application;

[0023] Fig. 5 is a pixel position correspondence calculation flowchart provided by an embodiment of the present application;

[0024] Fig. 6 is a schematic diagram of parameter data corresponding to front and rear cameras provided by an embodiment of the present application;

[0025] Fig. 7 is a schematic diagram of parameter data corresponding to left and right cameras provided by an embodiment of the present application;

[0026] Fig. 8 is a weight data acquisition flowchart provided by an embodiment of the present application;

[0027] Fig. 9 is a multiple region division flowchart provided by an embodiment of the present application;

[0028] Fig. 10 is a specific region division flowchart provided by an embodiment of the present application;

[0029] Fig. 11 is a weight data calculation flowchart provided by an embodiment of the present application;

[0030] Fig. 12 is a schematic diagram of weight data provided by an embodiment of the present application;

[0031] Fig. 13 is a panoramic image stitching model schematic diagram provided by an embodiment of the present application;

[0032] FIG. 14 is a schematic diagram of a panoramic image stitching model training process according to an embodiment of the present application;

[0033] FIG. 15 is a schematic diagram of a structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] The above and other advantages and features of the present application will become apparent from the following description of the embodiments, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the application. It is to be understood that the terminology used herein is for the purpose of describing the particular embodiments only and is not intended to be limiting of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application pertains.

[0035] It is to be understood that the terminology used herein is for the purpose of describing the particular embodiments only and is not intended to be limiting of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application pertains.

[0036] In the following description, numerous specific details are discussed to provide a thorough understanding of the embodiments of the application. One of ordinary skill in the related art will recognize, however, that the application can be practiced without one or more of the specific details. In some instances, well-known structures and devices are shown in block diagram form to avoid obscuring the application.

[0037] The flow diagrams and block diagrams in the drawings are schematic representations for purposes of illustration and do not necessarily depict the architectural, functional, or operational of all variations of the methods and computer program products disclosed herein. In this regard, each block in the flow diagrams or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions ("instructions"). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0038] Referring to FIG. 1, FIG. 1 is a kind of end-to-end panoramic image splicing method provided in an embodiment of the present application, comprising the following steps: S100, obtain the picture of vehicle multi-channel camera shooting.The panoramic camera of car is a kind of vehicle-mounted vision safety system, for collecting the picture around car body, by image processing means the fish-eye image collected is changed into the bird's-eye view picture without distortion, finally multiple images are spliced into a car bird's-eye view and are displayed in real time to driver, so as to reduce the traffic accident caused by driver's visual blind area.Panoramic image is generally obtained by splicing the picture shot by multiple cameras, the common camera scheme is four fish-eye cameras, of course, in order to obtain more accurate picture around vehicle, more cameras can be arranged in other schemes, in this step, first, the picture shot by all cameras needs to be obtained.

[0039] It can be understood that the above-mentioned picture can be RGB image data, YUV image data or other format image data.

[0040] The end-to-end panoramic image splicing method further comprises the following steps: S200, processing multiple pictures to obtain camera data.Because the specifications and quantity of multiple pictures are not very uniform, in order to adapt to more occasions, multiple pictures are processed here, so as to obtain camera data of uniform format, to facilitate subsequent model processing.

[0041] The end-to-end panoramic image splicing method further comprises the following steps: S300, inputting camera data, preset parameter data and weight data into the trained panoramic image splicing model to generate panoramic image graph, the parameter data is the coordinate mapping relationship of picture and panoramic image graph, and the weight data is the weight of picture in panoramic image graph.In order to realize end-to-end panoramic image splicing, preset parameter data and weight data are introduced in this embodiment, wherein the parameter data is the coordinate mapping relationship of picture and panoramic image graph, which is mainly to introduce the parameter characteristics of camera;The weight data is the weight of picture in panoramic image graph.Finally, camera data, parameter data and weight data are input into the trained panoramic image splicing model to obtain spliced panoramic image graph, and end-to-end panoramic image splicing is realized.

[0042] The above-mentioned steps are realized by end-to-end method, without intermediate conversion, and without brute force matching, camera data, parameter data and weight data are directly input into the trained panoramic image splicing model, and a panoramic image graph can be obtained;In the method, the parameters of the model are less, the inference time is less, a very large GPU computing power is not needed, the real-time performance is higher, and it is very suitable for embedded platform.

[0043] In an embodiment of the present application, step S100 comprises: acquiring four pictures taken at the same time by the front camera, the rear camera, the left camera and the right camera. As described above, in the most common panoramic image scheme at present, four fisheye cameras are generally included, which are generally referred to as the front camera, the rear camera, the left camera and the right camera according to their different positions on the vehicle. In the following detailed description of each embodiment, the four cameras are taken as examples for detailed description.

[0044] Referring to FIG. 2, in an embodiment of the present application, step S200 comprises: S201, performing reduction processing on the four pictures so that the length and width of the four pictures are reduced to half of the original picture; S202, splicing the four reduced pictures into one picture with the same size as the original picture according to a preset order to obtain camera data. In this embodiment, through the reduction and splicing processing on the original four pictures, a picture with the same size as the original picture can be obtained. For example, taking a 3 million pixel RGB picture as an example, the resolution of the pictures taken by the four cameras is 1920x1536, and the channel number is 3. First, the four pictures are scaled to a resolution of 960x768, and then splicing is performed.

[0045] The splicing process may, for example, place the four pictures of the left camera, the right camera, the front camera and the rear camera in the upper left, the upper right, the lower left and the lower right positions respectively for splicing. The splicing order can be arbitrary, but it needs to be noted that after the splicing order is determined, the same order needs to be used for splicing during model training and panoramic image splicing. The camera data after splicing is shown in FIG. 3.

[0046] Referring to FIG. 4, in an embodiment of the present application, the preset parameter data is obtained through the following steps: S301, acquiring the inverse mapping relationship from the world coordinates to the camera coordinates (also referred to as camera coordinates) according to the intrinsic and extrinsic parameters of the multi-camera. For example, the inverse mapping relationship can be acquired through the following steps: first, calculating the rotation matrix R 3×3 and the translation matrix T 3×1 according to the camera extrinsic parameters, and then calculating the distortion LUT parameter table based on the intrinsic parameters of the camera and . The specific formula is as follows:

[0047] In the formula, [x w ,y w ,z w ] T is the world coordinate, and [u,v] T ​wherein r represents the arc length of the non-distorted point, and r' represents the arc length of the distorted point. Understandably, when there are multiple cameras, each camera corresponds to an inverse mapping relationship.

[0048] Referring to FIG. 5, the preset parameter data is further obtained by the following steps: S302, calculating the correspondence between the world coordinates and the pixel positions of the pictures captured by the multiple cameras according to the inverse mapping relationship. After obtaining the inverse mapping relationship, the correspondence can be further obtained. For example, in an embodiment of the present application, step S302 includes: S3021, for the world coordinates within the camera shooting range, calculating the first correspondence between the world coordinates and the pixel positions of the pictures captured by the cameras according to the inverse mapping relationship; S3022, for the world coordinates beyond the camera shooting range, marking the corresponding pixel positions as (0, 0) to obtain the second correspondence between the world coordinates and the pixel positions of the pictures captured by the cameras; S3023, combining the first correspondence and the second correspondence to obtain a mapping table matrix to represent the correspondence between the world coordinates and the pixel positions of the pictures captured by the multiple cameras; wherein the mapping table matrix corresponds to the cameras.

[0049] The above steps will be further described in combination with specific examples. In order to align the camera data size, 1920x1536 is taken as the grid range, each grid point is sampled by 16mm, and the corresponding field of view range is 30.72m x 24.576m. A world coordinate system is established, for example, the center point of the rear axle of the vehicle can be taken as the coordinate origin, the width direction of the vehicle can be taken as the X direction, the length direction of the vehicle can be taken as the Y direction, and the height direction of the vehicle can be taken as the Z direction, so that the coordinate information (x w ,y w ,z w ) of each pixel position in the world coordinates in the world coordinate system can be obtained. Taking the left camera as an example, the pixel position of each pixel position in the world coordinates in the picture captured by the left camera can be obtained according to the coordinate information (x w ,y w ,z w ) and the inverse mapping relationship, and the pixel position of the pixel position in the world coordinates beyond the shooting range of the left camera in the picture captured by the left camera is marked as (0, 0), thereby forming a 1920x1536 size mapping table matrix, each element of which corresponds to the pixel position of the picture captured by the left camera. For the other three cameras, three 1920x1536 size mapping table matrices can be obtained in the same way, so that four mapping table matrices can be obtained in total.

[0050] Please refer to FIG. 6 and FIG. 7, since the elements in the mapping table matrix are pixel positions, the element value range is 0-1920, the span is relatively large, which is not conducive to subsequent calculation, and thus normalization processing is needed, for example, normalization to 0-1 or other ranges. In the foregoing scheme, the size of the camera data is 1920x1536x3, and the value of each pixel is 0-255, and thus in an embodiment of the present application, for example, the four mapping table matrices can be normalized to 0-255. In order to facilitate understanding, the four normalized mapping table matrices are combined two by two to obtain two visualized pictures, for example, the first picture can be obtained by combining the normalized mapping table matrices corresponding to the front camera and the rear camera, wherein the R channel of the first picture is the x coordinate of the mapping table matrix corresponding to the front camera, the G channel is the y coordinate of the mapping table matrix corresponding to the front camera, the B channel is the x coordinate of the mapping table matrix corresponding to the rear camera, and the A channel is the y coordinate of the mapping table matrix corresponding to the rear camera. The first picture is shown in FIG. 6. The second picture can be obtained by combining the normalized mapping table matrices corresponding to the left camera and the right camera, wherein the R channel of the second picture is the x coordinate of the mapping table matrix corresponding to the left camera, the G channel is the y coordinate of the mapping table matrix corresponding to the left camera, the B channel is the x coordinate of the mapping table matrix corresponding to the right camera, and the A channel is the y coordinate of the mapping table matrix corresponding to the right camera. The second picture is shown in FIG. 7. The first picture and the second picture jointly constitute the parameter data.

[0051] Please refer to FIG. 8, in an embodiment of the present application, the preset weight data is obtained by the following steps: S310, according to the positional relationship of the multi-camera, the panoramic image to be generated is divided into multiple regions; S320, according to the relationship between the region and the shooting range of the camera, the weight data is obtained. By introducing the concept of multiple regions, the calculation of the weight data can be simplified.

[0052] Please refer to FIG. 9, in an embodiment of the present application, the step S310 includes: S311, according to the positional relationship of the multi-camera and the vehicle, the image size of the panoramic image to be generated is determined; S312, according to the image size, according to the positional relationship of the multi-camera and the vehicle, the panoramic image to be generated is divided into multiple regions. In this embodiment, a specific region division method is provided, that is, the panoramic image is divided by the positional relationship of the multi-camera and the vehicle. The reason for such processing is that in the panoramic image, the cameras are all installed around the vehicle and face outward, and thus the weight matrix corresponding to the vehicle region and the non-vehicle region must be different. At the same time, the positional relationship of the multi-camera directly affects the weight of each camera in the non-vehicle region.

[0053] Please refer to FIG. 10, in a specific embodiment of the present application, step S312 comprises: S3121, setting the region corresponding to the vehicle in the panoramic image to be generated as the vehicle region, for example, in the weight data shown in FIG. 12, the vehicle region is the middle region; S3122, according to the shooting range of the multi-camera, setting the region covered only by the shooting range of one camera in the panoramic image to be generated as the non-overlapping region, for example, in the weight data shown in FIG. 12, the non-overlapping region is the front, rear, left and right four regions, which correspond to the front camera, rear camera, left camera and right camera respectively; S3123, according to the shooting range of the multi-camera, setting the region covered by the shooting range of at least two cameras in the panoramic image to be generated as the overlapping region, for example, in the weight data shown in FIG. 12, the overlapping region is the front-left, front-right, rear-left and rear-right four regions, the front-left region can be shot by both the front camera and the left camera, so the front-left region corresponds to the front camera and the left camera, and other regions also correspond to two cameras.

[0054] Please refer to FIG. 11 and FIG. 12, in a specific embodiment of the present application, step S320 comprises: S321, when the region is the vehicle region, the weight data of all pixel points located in the vehicle region is 0, for example, the initial weight of each pixel point in the vehicle region can be recorded as [0, 0, 0, 0], and the four 0s correspond to the weights of the four cameras respectively.

[0055] Step S320 further comprises: S322, when the region is the non-overlapping region, the weight data of all pixel points located in the non-overlapping region is related to the camera corresponding to the region and has the same value. The initial weight of the non-overlapping region can be, for example: the initial weight of the front region is recorded as [1, 0, 0, 0]; the initial weight of the rear region is recorded as [0, 1, 0, 0]; the initial weight of the left region is recorded as [0, 0, 1, 0]; and the initial weight of the right region is recorded as [0, 0, 0, 1]. Taking the front region as an example, the region only appears in the shooting range of the front camera, and other cameras cannot shoot the front region, so the weight representing the front camera is recorded as 1, and the others are recorded as 0.

[0056] The step S320 further comprises: S323, when the region is an overlapping region, the weight data of all pixel points in the overlapping region is related to the specific position of the pixel point and the contribution degree of the multiple cameras corresponding to the region. The overlapping region comprises a front-left region, a front-right region, a back-left region and a back-right region, and the initial weight corresponding to the regions is, for example, the initial weight of the front-left region is denoted as [a, 0, 1-a, 0], the initial weight of the front-right region is denoted as [a, 0, 0, 1-a], the initial weight of the back-left region is denoted as [0, a, 1-a, 0], and the initial weight of the back-right region is denoted as [0, a, 0, 1-a]. Taking the front-left region as an example, since the back camera and the right camera cannot shoot the region, the 2nd and 4th numbers are 0, and assuming that the contribution degree of the front camera in the front-left region is a, then the contribution degree of the left camera is 1-a, so the initial weight of the front-left region is [a, 0, 1-a, 0], and the weight of the other overlapping regions can be calculated in the same way.

[0057] The contribution degree can be understood as that the closer the corresponding position of a certain pixel point of the weight data in the picture shot by the camera to the edge, the lower the contribution degree. In actual calculation, a can be calculated according to the relative distance or angle of each pixel point to the corresponding two regions.

[0058] For the front-left region, which is the overlapping region of the front camera and the left camera, when calculating the specific a, taking the angle as an example, the initial weight of multiple positions corresponding to the same angle should be the same. Please refer to FIG. 12, in a specific embodiment of the present application, the weight value a is related to the angle θ between each actual position and the front region, and for example, the weight value a can be calculated in the following way: the weight value a is 1.0 between 0 and θ1, linearly changes from 1 to 0.5 between θ1 and θ2, linearly changes from 0.5 to 0 between θ2 and θ3, and is 0 between θ3 and π / 2. Wherein, θ1, θ2 and θ3 can be set to 30°, 45° and 60° according to experience. It can be understood that when the definition of the angle θ is different, the calculation method of the weight value a is also different, and the specific calculation method and the angle value setting in the example are only for reference.

[0059] It can be understood that the value of the initial weight is 0-1, in order to maintain consistency with the camera data and the parameter data, it can also be normalized to the range of 0-255, and after normalization, the weight data can be mapped to a third picture, wherein the R channel of the third picture represents the normalized weight value of the front camera, the G channel represents the normalized weight value of the back camera, the B channel represents the normalized weight value of the left camera, and the A channel represents the normalized weight value of the right camera, and the third picture is shown in FIG. 12.

[0060] It can be understood that the above parameter data and weight data are generated only once at initialization, for example, can be generated when the vehicle is manufactured, and then can be saved as preset parameter data and weight data, which can be directly read in subsequent use.

[0061] It can be understood that if the sizes of the camera data, the parameter data, and the weight data are different, size adjustment can be performed first. The adjusted three kinds of data can be directly input into the trained panoramic image stitching model, or can be input into the model after combination. The combination method can be, for example, matrix addition, matrix multiplication, or matrix dimension expansion, and the like.

[0062] Please refer to FIG. 13. In a specific embodiment of the present application, step S300 includes: inputting the camera data, the preset parameter data, and the weight data into the encoder (Encode module) of the panoramic image stitching model to extract picture features; inputting the extracted picture features into the decoder (Decode module) of the panoramic image stitching model to analyze the picture features; and inputting the analyzed picture features into the full convolution network layer (the rightmost CNN in FIG. 13) of the panoramic image stitching model to generate a panoramic image of a preset size. In this embodiment, the panoramic image stitching model includes the encoder, the decoder, and the full convolution network layer, wherein the encoder and the decoder are mainly used to extract and analyze picture features, and the full convolution network layer is mainly used to adjust the size of the panoramic image output by the model, thereby outputting panoramic images of different fields of view.

[0063] It can be understood that the encoder and the decoder can be implemented using a network such as a Transformer or a Resnet, or can be composed of convolution layers. For example, in the panoramic image stitching model shown in FIG. 13, a three-layer convolutional neural network is used for the down-sampling Encode module and a three-layer convolutional neural network is used for the up-sampling Decode module. The former Encode module is used to extract image features, and the latter Decode module is used to analyze image features. Finally, a full convolution network (i.e., CNN) is used to generate a panoramic image. Using a seven-layer convolutional neural network has fewer network parameters and less inference time, and does not require very high GPU computing power, while having high real-time performance.

[0064] It can be understood that after the panoramic image stitching model is constructed, the model needs to be trained. During training, a traditional method of optical flow and feature point matching can be used to obtain a high-precision panoramic image as GroundTrue, and then a loss function of GroundTrue and model output is minimized. The training process is shown in FIG. 14. During training, for example, mean square error (MSE) can be used as the loss function, and the error between the high-precision panoramic image and the panoramic image output by the panoramic image stitching model is minimized.

[0065] It can be understood that a multi-scale panoramic image can be used to train a panoramic image stitching model of different fields of view, and a panoramic image of a certain fixed scale can also be used to train a panoramic image stitching model. For the former model, a multi-scale training set needs to be prepared, and the training process is also longer, but the model trained can well adapt to panoramic images of different fields of view. For the latter model, only a single scale training set needs to be prepared, and the model finally trained is only suitable for panoramic images of this size.

[0066] Referring to FIG. 15, an electronic device provided by an embodiment of the present application includes a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory, so as to implement the panoramic image stitching method in any one of the above embodiments.

[0067] In a specific embodiment of the present application, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is used to make a computer execute the panoramic image stitching method in any one of the above embodiments.

[0068] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought of the present application should be covered by the claims of the present application.

Claims

1. An end-to-end panoramic image stitching method, characterized by, The method comprises the following steps: acquiring pictures captured by multiple cameras of a vehicle; processing multiple pictures to obtain camera data; inputting the camera data, preset parameter data and weight data into a trained panoramic image stitching model to generate a panoramic image, wherein the parameter data is a coordinate mapping relationship between the pictures and the panoramic image, and the weight data is a weight of the pictures in the panoramic image.

2. The end-to-end panoramic image stitching method of claim 1, wherein, The preset parameter data is obtained by the following steps: obtaining an inverse mapping relationship from world coordinates to camera coordinates according to intrinsic and extrinsic parameters of the multiple cameras; calculating a corresponding relationship between the world coordinates and pixel positions of the pictures captured by the multiple cameras according to the inverse mapping relationship; obtaining the parameter data according to the corresponding relationship. 3.The end-to-end panoramic image stitching method of claim 2, wherein, The step of calculating the corresponding relationship between the world coordinates and the pixel positions of the pictures captured by the multiple cameras according to the inverse mapping relationship comprises: for each world coordinate within a camera capturing range, calculating a first corresponding relationship between the world coordinate and a pixel position of a picture captured by the camera according to the inverse mapping relationship; for a world coordinate beyond the camera capturing range, marking the corresponding pixel position as (0, 0) to obtain a second corresponding relationship between the world coordinate and the pixel position of the picture captured by the camera; combining the first corresponding relationship and the second corresponding relationship to obtain a mapping table matrix to represent the corresponding relationship between the world coordinates and the pixel positions of the pictures captured by the multiple cameras; wherein the mapping table matrix corresponds to the camera.

4. The end-to-end panoramic image stitching method of claim 3, wherein, The step of obtaining the parameter data according to the corresponding relationship comprises: combining multiple mapping table matrices and performing normalization to obtain the parameter data. 5.The end-to-end panoramic image stitching method of claim 1, wherein, The preset weight data is obtained by the following steps: dividing the panoramic image to be generated into multiple regions according to a positional relationship of the multiple cameras; obtaining the weight data according to a relationship between the regions and camera capturing ranges.

6. The end-to-end panoramic image stitching method of claim 5, wherein, The step of dividing the panoramic image to be generated into multiple regions according to a positional relationship of the multiple cameras comprises: determining an image size of the panoramic image to be generated according to the positional relationship of the multiple cameras and the vehicle; dividing the panoramic image to be generated into multiple regions according to the positional relationship of the multiple cameras and the vehicle based on the image size.

7. The end-to-end panoramic image stitching method of claim 6, wherein, The step of dividing the panoramic image to be generated into multiple regions according to the positional relationship of the multiple cameras and the vehicle based on the image size comprises: setting a region corresponding to the vehicle in the panoramic image to be generated as a vehicle region; setting a region covered by only one camera in the panoramic image to be generated as a non-overlapping region according to camera capturing ranges of the multiple cameras; setting a region covered by at least two cameras in the panoramic image to be generated as an overlapping region according to the camera capturing ranges of the multiple cameras. 8.The end-to-end panoramic image stitching method of claim 7, wherein, The step of obtaining the weight data according to the relationship between the region and the shooting range of the camera comprises: When the region is a vehicle region, the weight data of all pixel points in the vehicle region is 0; When the region is a non-overlapping region, the weight data of all pixel points in the non-overlapping region is related to the camera corresponding to the region and the value is equal; When the region is an overlapping region, the weight data of all pixel points in the overlapping region is related to the specific position of the pixel points and the contribution degree of the multiple cameras corresponding to the region.

9. An electronic device, comprising: The device comprises a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute the computer program stored in the memory to realize the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer program is stored on the device and is used to make the computer execute the method according to any one of claims 1-8.

Images