Model training method and image generation method, device and equipment, and storage medium

Abstract

The application relates to a model training method and an image generation method, device and equipment and a storage medium. The technical scheme of the model training method comprises the following steps: acquiring a low-resolution image sequence captured by a vehicle camera at multiple viewing angles, the low-resolution image sequence comprising multiple low-resolution images; generating a target frame low-definition bird's-eye view and a true value high-resolution bird's-eye view according to the multiple low-resolution images; training an adversarial generation network based on the target frame low-definition bird's-eye view and the true value high-resolution bird's-eye view to obtain a detail compensation model. The detail compensation model can convert a low-definition bird's-eye view into a high-definition bird's-eye view, provide an input image with high availability for an automatic parking function, improve the utilization rate of the low-definition bird's-eye view, and obtain more reliable details, which is beneficial to improving the accuracy of a detection result of any kind of perception detection algorithm based on a bird's-eye view puzzle as input.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a model training method, image generation method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of the automotive industry, automotive driver assistance functions are becoming increasingly sophisticated. Automatic parking is one such function, which uses real-time images captured by cameras to detect and identify parking space lines, and then completes automatic parking through a parking system.

[0003] Currently, the automatic parking system typically uses surround-view fisheye cameras to stitch together images from a bird's-eye view (BEV) perspective using an Around View Monitor (AVM) system. However, due to the unique imaging characteristics of fisheye lenses, the image graininess and noise increase as distance from the vehicle increases in the stitched BEV image from the AVM, resulting in a less clear image and failing to provide a usable input image for the automatic parking function. Summary of the Invention

[0004] Based on this, this application provides a model training method, image generation method, apparatus, device, and storage medium that can generate high-resolution bird's-eye view images, providing highly usable input images for automatic parking functions.

[0005] Firstly, a model training method is provided, which includes:

[0006] Acquire a sequence of low-resolution images captured by the vehicle's camera from multiple perspectives. The sequence of low-resolution images includes multiple frames of low-resolution images.

[0007] Based on multiple low-resolution images, generate a low-resolution bird's-eye view of the target frame and a true high-resolution bird's-eye view;

[0008] An adversarial generative network is trained using low-resolution bird's-eye view of the target frame and high-resolution ground truth bird's-eye view to obtain a detail compensation model.

[0009] According to one achievable method in an embodiment of this application, generating a low-resolution bird's-eye view of a target frame based on multiple low-resolution images includes:

[0010] By stitching together the target low-resolution image from multiple low-resolution images, a low-resolution bird's-eye view of the target frame is obtained. The target low-resolution image is a low-resolution image located in the same frame from multiple low-resolution images from different perspectives.

[0011] According to one feasible method in an embodiment of this application, generating a true high-resolution bird's-eye view based on multiple frames of low-resolution images includes:

[0012] Based on the position of the target frame low-resolution image in the multiple low-resolution images, determine the preparation low-resolution image;

[0013] Align the image features of two adjacent low-resolution images in the prepared low-resolution image to obtain the first low-resolution image feature;

[0014] Extract and fuse the features of the first low-resolution image to obtain the features of the second low-resolution image;

[0015] Based on the features of the second low-resolution image, a high-resolution image of the target frame is generated.

[0016] By stitching together the high-resolution images of the target frame, a true high-resolution bird's-eye view is obtained.

[0017] According to one feasible method in an embodiment of this application, determining a preliminary low-resolution image based on the position of the target frame low-resolution image among multiple low-resolution images includes:

[0018] Based on the position of the target frame low-resolution image in the multi-frame low-resolution images, a preliminary low-resolution image is determined based on a sliding window of a preset size, with the target frame low-resolution image located in the middle of the sliding window.

[0019] According to one feasible method in an embodiment of this application, image features of two adjacent low-resolution images in a prepared low-resolution image are aligned to obtain a first low-resolution image feature, including:

[0020] Obtain the pixel displacement vectors of two adjacent low-resolution frames in the prepared low-resolution image;

[0021] Based on the pixel displacement vector, align the image features of two adjacent low-resolution images in the prepared low-resolution image to obtain the first low-resolution image feature.

[0022] According to one feasible method in an embodiment of this application, obtaining the pixel displacement vectors of two adjacent low-resolution images in a prepared low-resolution image includes:

[0023] An optical flow estimation algorithm is used to obtain the pixel displacement vectors of two adjacent low-resolution frames in the prepared low-resolution image.

[0024] According to one feasible method in an embodiment of this application, obtaining the pixel displacement vectors of two adjacent low-resolution images in a prepared low-resolution image includes:

[0025] The pixel displacement vectors of two adjacent low-resolution frames in the preparatory low-resolution image are obtained by using a deformable convolutional network.

[0026] According to one achievable method in an embodiment of this application, the adversarial generative network includes a generator and a discriminator. The adversarial generative network is trained based on a low-resolution bird's-eye view of the target frame and a ground truth high-resolution bird's-eye view to obtain a detail compensation model, including:

[0027] Based on the low-resolution bird's-eye view of the target frame, a predictive high-resolution bird's-eye view is generated using a generator.

[0028] Based on the true high-definition bird's-eye view, the type information of the predicted high-definition bird's-eye view is determined by a discriminator;

[0029] Based on the type information, determine the detailed compensation model.

[0030] According to one achievable method in an embodiment of this application, the type information includes a generator generation graph and a truth generation graph. Based on the type information, a detail compensation model is determined, including:

[0031] When the type information is a generator-generated image, a predicted high-resolution bird's-eye view is generated by the generator based on the target low-resolution bird's-eye view.

[0032] The generator is identified as a detail compensation model when the discriminator determines that the type information of the predicted high-definition bird's-eye view is a true value generated image and the recognition rate is greater than a preset threshold.

[0033] Secondly, a computer device is provided, comprising:

[0034] At least one processor; and

[0035] A memory that is communicatively connected to at least one processor; wherein,

[0036] The memory stores computer instructions that can be executed by at least one processor to enable the at least one processor to perform the methods involved in the first aspect above.

[0037] Thirdly, a computer-readable storage medium is provided having computer instructions stored thereon, characterized in that the computer instructions are used to cause a computer to perform the methods involved in the first aspect above.

[0038] The aforementioned model training method, along with its related devices, computer equipment, and storage media, acquires a sequence of low-resolution images captured by a vehicle camera from multiple perspectives. This sequence comprises multiple frames of low-resolution images. Based on these frames, a target frame low-resolution bird's-eye view and a ground-value high-resolution bird's-eye view are generated. An adversarial generative network is trained using both the target frame low-resolution bird's-eye view and the ground-value high-resolution bird's-eye view to obtain a detail compensation model. This detail compensation model can convert low-resolution bird's-eye views into high-resolution bird's-eye views, providing highly usable input images for automatic parking functions and improving the utilization rate of low-resolution bird's-eye views.

[0039] Fourthly, an image generation method is provided, the method comprising:

[0040] Acquire a driving environment map of the vehicle, which includes image sequences from multiple perspectives;

[0041] Generate low-resolution bird's-eye view based on image sequences from multiple perspectives;

[0042] A high-resolution bird's-eye view is generated based on a low-resolution bird's-eye view and a pre-trained detail compensation model, which is trained by any of the feasible methods of the first aspect.

[0043] According to one feasible method in an embodiment of this application, a low-resolution bird's-eye view is generated based on an image sequence from multiple perspectives, including:

[0044] By stitching together image sequences from multiple perspectives, a low-resolution bird's-eye view is obtained.

[0045] According to one feasible method in an embodiment of this application, image sequences from multiple perspectives are stitched together to obtain a low-resolution bird's-eye view, including:

[0046] Based on the camera's intrinsic parameters and distortion coefficients of the captured image sequence, distortion correction processing is performed on the image sequence from multiple viewpoints to obtain the processed image sequence.

[0047] The processed image sequence is jointly calibrated based on preset calibration parameters to obtain the projection matrix;

[0048] The processed image sequence is transformed by projection matrix to obtain at least one initial bird's-eye view.

[0049] By stitching and merging at least one initial bird's-eye view frame, a low-resolution bird's-eye view can be obtained.

[0050] According to one achievable method in an embodiment of this application, the detail compensation model includes an encoder and a decoder, and generates a high-resolution bird's-eye view based on a pre-trained detail compensation model from a low-resolution bird's-eye view, including:

[0051] The first feature map is generated by an encoder based on the low-resolution bird's-eye view;

[0052] A high-resolution bird's-eye view is generated using a decoder based on the first feature map.

[0053] According to one possible implementation in an embodiment of this application, the encoder includes a first convolutional layer and a downsampling layer. Based on a low-resolution bird's-eye view, the encoder generates a first feature map, including:

[0054] Based on the low-resolution bird's-eye view, an initial feature map is generated through the first convolutional layer;

[0055] Based on the initial feature map, a first feature map is generated through a downsampling layer.

[0056] According to one possible implementation in an embodiment of this application, the decoder includes a second convolutional layer and an upsampling layer, and generates a high-resolution bird's-eye view based on the first feature map, including:

[0057] Based on the first feature map, a combined feature map is generated through the second convolutional layer;

[0058] A high-resolution bird's-eye view is generated by upsampling the combined feature map.

[0059] Fifthly, a computer device is provided, comprising:

[0060] At least one processor; and

[0061] A memory that is communicatively connected to at least one processor; wherein,

[0062] The memory stores computer instructions that can be executed by at least one processor, which enables the at least one processor to perform the methods involved in the fourth aspect above.

[0063] In a sixth aspect, a computer-readable storage medium is provided, having stored thereon computer instructions, characterized in that the computer instructions are used to cause a computer to perform the methods involved in the fourth aspect above.

[0064] The aforementioned image generation method, along with its related apparatus, computer equipment, and storage medium, acquires a driving environment map of the vehicle. Based on image sequences from multiple perspectives within the driving environment map, it generates a low-resolution bird's-eye view. This low-resolution bird's-eye view is then input into a pre-trained detail compensation model to obtain a high-resolution bird's-eye view. Because the detail compensation model improves the imaging effect and accuracy of the low-resolution bird's-eye view, it provides a highly usable input image for automatic parking functions, increasing the utilization rate of the low-resolution bird's-eye view. Furthermore, it acquires more reliable details, which is beneficial for improving the accuracy of any subsequent perception detection algorithm based on bird's-eye view mosaicking as input. Attached Figure Description

[0065] Figure 1 This is an application environment diagram of the image generation method in one embodiment;

[0066] Figure 2 This is a flowchart illustrating a model training method in one embodiment;

[0067] Figure 3 This is a flowchart illustrating the steps for generating a true high-resolution bird's-eye view in one embodiment;

[0068] Figure 4 This is a flowchart illustrating the training steps of an adversarial generative network in one embodiment.

[0069] Figure 5 This is a flowchart illustrating an image generation method in one embodiment;

[0070] Figure 6 This is a schematic diagram of the AVM stitching algorithm in one embodiment;

[0071] Figure 7 This is a flowchart illustrating the image sequence stitching steps in one embodiment;

[0072] Figure 8 This is a flowchart illustrating the steps involved in generating a high-resolution bird's-eye view in one embodiment.

[0073] Figure 9 This is a schematic structural diagram of a computer device in one embodiment. Detailed Implementation

[0074] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the scope of the present application.

[0075] The model training method and image generation method provided in this application can be applied to... Figure 1 The vehicle 100 shown may include an in-vehicle terminal 110 and an in-vehicle camera 120. The in-vehicle terminal 110 includes at least one memory and at least one processor. The at least one memory stores a computer program that, when executed by the at least one processor, performs a model training method and an image generation method according to exemplary embodiments of this application. Here, the in-vehicle terminal 110 is not necessarily a single electronic device, but may be any collection of devices or circuits capable of executing the aforementioned computer program individually or in combination.

[0076] In the vehicle terminal 110, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0077] In the vehicle-mounted terminal 110, the processor can run a computer program stored in the memory. This computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...). These one or more modules / units are stored in the memory 20 and executed by the processor 10 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, describing the execution process of the computer program in the terminal device. For example, the detail compensation model in this embodiment can be one of these modules / units.

[0078] Memory can be integrated with the processor; for example, RAM or flash memory can be housed within an integrated circuit microprocessor. Alternatively, memory can comprise a separate device, such as an external disk drive, storage array, or other storage device that can be used by any database system. Memory and processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the memory.

[0079] In addition, the vehicle terminal 110 may also include a display device (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the vehicle terminal 110 can be interconnected via a bus and / or network.

[0080] The vehicle-mounted camera 120 can be a fisheye camera. The number of vehicle-mounted cameras 120 and their installation positions on the vehicle 100 can be determined according to actual needs. Generally, it can include four fisheye cameras, namely a front-view fisheye camera, a rear-view fisheye camera, a left-view fisheye camera, and a right-view fisheye camera, which are used to collect environmental information in the four directions of the vehicle, front, rear, left, and right.

[0081] Figure 2 A flowchart illustrating a model training method provided in this application embodiment, the method can be performed by, for example... Figure 1 The vehicle-mounted terminal 110 in the system shown executes this. For example... Figure 2 As shown, the method may include the following steps:

[0082] S210 acquires a sequence of low-resolution images captured by the vehicle's camera from multiple perspectives.

[0083] Image generation methods can be understood as an application of detail compensation models. Therefore, those skilled in the art will understand that the training of detail compensation models and the application of detail compensation models are related. That is, detail compensation model training methods and image generation methods are related at the principle or step level. Some of the same terms that have appeared in the previous text will not be explained again in the following text, and some of the same principles that have appeared in the previous text will not be explained in too much detail in the following text.

[0084] The low-resolution image sequence includes multiple low-resolution images, each consisting of at least one low-resolution image captured from each viewpoint. These low-resolution images are images of the surrounding driving environment captured by the vehicle's cameras from multiple viewpoints. The surrounding driving environment can be the driving environment within a preset distance of the vehicle, which can be determined based on actual business needs, product requirements, or application scenarios.

[0085] Low-resolution image sequences can be obtained from historical driving data stored in the vehicle, or by driving the vehicle to a specific environment to collect low-resolution image sequences in real time. The specific environment can be a relatively representative environment, such as an environment where the user's vehicle frequently passes through or an environment with high traffic volume.

[0086] S220 generates a low-resolution bird's-eye view of the target frame and a true high-resolution bird's-eye view based on multiple low-resolution images.

[0087] The target frame's low-resolution bird's-eye view is obtained by stitching together low-resolution images located within the target frame from multiple low-resolution images viewed from different perspectives. When calculating the low-resolution bird's-eye view and high-resolution image of the target frame, the other frames serve as reference frames. Any low-resolution image from the multiple low-resolution images can become the target frame's low-resolution image. The AVM stitching method can be used to stitch together the low-resolution images located within the target frame from multiple low-resolution images viewed from different perspectives to obtain the target frame's low-resolution bird's-eye view. Specifically, this may include distortion correction, joint calibration, projection transformation, and stitching fusion. The method for stitching the target low-resolution image is the same as the stitching method in S520; the specific implementation process of the stitching method will be described in detail later.

[0088] The true high-resolution bird's-eye view is obtained by stitching together high-resolution images of the target frame. These target frame high-resolution images are generated from a preset number of low-resolution images at each viewpoint determined by the target frame. The preset number can be set according to computational needs; the larger the preset number, the higher the accuracy of the target frame high-resolution image. The target frame high-resolution image can be generated solely from the low-resolution images of the target frame, in which case the preset number is 1. Alternatively, it can be generated from the target frame and its preceding and following low-resolution images. Ideally, the number of preceding and following frames should be the same; in this case, the preset number can be 2N+1 or N+1, where N is the number of preceding or following frames of the target frame.

[0089] When the preset number is greater than 1, a multi-frame super-resolution algorithm can be used to align the image features of multiple low-resolution images. Then, the aligned frames are upsampled and downsampled. Finally, the difference information of the aligned features between the frames is extracted, and the feature information of multiple frames is fused temporally to obtain higher-order features. Based on the higher-order features, upsampling and convolution processing are performed to obtain the target frame high-resolution image. The difference information can include the pixel features that are different at the same position between two adjacent low-resolution images.

[0090] S230 trains an adversarial generative network based on a low-resolution bird's-eye view of the target frame and a ground truth high-resolution bird's-eye view to obtain a detail compensation model.

[0091] The Generative Adversarial Network (GAN) consists of a generator and a discriminator. A low-resolution bird's-eye view of the target frame is input into the generator to produce a high-resolution bird's-eye view. The ground truth high-resolution bird's-eye view is used as the reference bird's-eye view for the discriminator. The generated high-resolution bird's-eye view is then input into the discriminator, which judges the similarity between the stitched high-resolution bird's-eye view and the generated high-resolution bird's-eye view to determine whether the input high-resolution bird's-eye view is stitched together or generated. When the discriminator determines that the input high-resolution bird's-eye view is stitched together and the recognition rate exceeds a certain value, the model training process ends, and the generator is used as a detail compensation model.

[0092] As can be seen, by acquiring a sequence of low-resolution images captured by the vehicle's camera from multiple perspectives, including multiple frames of low-resolution images, a target frame low-resolution bird's-eye view and a ground-value high-resolution bird's-eye view are generated based on these multiple frames. An adversarial generative network is then trained using the target frame low-resolution bird's-eye view and the ground-value high-resolution bird's-eye view to obtain a detail compensation model. This detail compensation model can convert low-resolution bird's-eye views into high-resolution bird's-eye views, providing highly usable input images for the automatic parking function and improving the utilization rate of low-resolution bird's-eye views.

[0093] In one feasible approach, the step of generating a low-resolution bird's-eye view of the target frame from multiple low-resolution images in S220 includes:

[0094] By stitching together low-resolution images of the target from multiple low-resolution images, a low-resolution bird's-eye view of the target frame is obtained.

[0095] The target low-resolution image is a low-resolution image located in the same frame among multiple low-resolution images from different viewpoints. For example, multiple viewpoints include the front view, the back view, the left view, and the right view, and the target frame is the 7th frame. The target low-resolution image includes the low-resolution image of the 7th frame in the multiple low-resolution images from the front view, the 7th frame in the multiple low-resolution images from the back view, the 7th frame in the multiple low-resolution images from the left view, and the 7th frame in the multiple low-resolution images from the right view.

[0096] The AVM stitching method can be used to stitch together low-resolution target images to obtain a low-resolution bird's-eye view of the target frame. This process can include distortion correction, joint calibration, projection transformation, and stitching fusion. The method for stitching the low-resolution target image is the same as the stitching method in S520, and the specific implementation process will be described in detail later.

[0097] In one feasible approach, the step in S220 of generating a ground-value high-resolution bird's-eye view based on multiple frames of low-resolution images is as follows: Figure 3 As shown, it may include:

[0098] S221, determine the preparation low-resolution image based on the position of the target frame low-resolution image in the multi-frame low-resolution images.

[0099] Here, the target frame low-resolution image is the low-resolution image corresponding to the target frame in a multi-frame low-resolution image set, and the position of the target frame low-resolution image within the multi-frame low-resolution image set is its frame order. The target frame can be selected according to computational needs and is not limited here.

[0100] Specifically, based on the position of the target frame low-resolution image among multiple low-resolution images, a preliminary low-resolution image is determined using a sliding window of a preset size, with the target frame low-resolution image located in the middle of the sliding window.

[0101] The size of the sliding window determines the number of low-resolution images in the prepared low-resolution image set. The sliding window consists of 2N+1 windows, where N can be chosen according to computational needs, and generally, the larger the better. The target frame low-resolution image is located in the middle of the sliding window, with N windows on each side of the middle position.

[0102] If there are N low-resolution images before and after the target frame's low-resolution image location, then the prepared low-resolution images consist of 2N+1 low-resolution images. If there are N low-resolution images before (or after) the target frame's low-resolution image location, and M low-resolution images after (or before) the target frame's low-resolution image location, then the prepared low-resolution images consist of N+M+1 low-resolution images. Where N>M. If the target frame's low-resolution image is located in the first or last frame of a multi-frame low-resolution image set, and there are N low-resolution images before (or after) the target frame's low-resolution image location, but no low-resolution images after (or before) the target frame's low-resolution image location, then the prepared low-resolution images consist of N+1 low-resolution images.

[0103] S222, Align the image features of two adjacent low-resolution images in the prepared low-resolution image to obtain the first low-resolution image features.

[0104] Because pixels in adjacent low-resolution frames shift and deform over time, differences occur at the same positions in adjacent low-resolution frames, resulting in ghosting when they are superimposed. Therefore, it is necessary to align the image features of adjacent low-resolution frames. The first low-resolution image feature is the aligned feature of the prepared low-resolution image.

[0105] When aligning the image features of two adjacent low-resolution images, the alignment starts from the target frame. Taking the target frame as frame T as an example, first align frames T-1 and T+1 with frame T, then align frame T-2 with frame T-1, and then align frame T+2 with frame T+1, so that the low-resolution images of the prepared low-resolution images are as similar as possible to the low-resolution images of the target frames.

[0106] Specifically, obtain the pixel displacement vectors of two adjacent low-resolution images in the prepared low-resolution image;

[0107] Based on the pixel displacement vector, align the image features of two adjacent low-resolution images in the prepared low-resolution image to obtain the first low-resolution image feature.

[0108] In this context, the pixel displacement vector is the motion vector of pixels in the later frame relative to the previous frame in two adjacent low-resolution image frames. One feasible approach is to use an optical flow estimation algorithm to obtain the pixel displacement vectors of two adjacent low-resolution image frames in the prepared low-resolution image. The optical flow estimation algorithm estimates the motion vectors between two consecutive image frames; optical flow is the motion vector between the two frames. The optical flow estimation algorithm determines the positional change of the same pixel in two image frames; this change is the displacement vector, which is the optical flow of the pixel. This can be achieved by first upsampling and downsampling the two adjacent low-resolution image frames, then inputting them into a pre-trained optical flow network to obtain the optical flow residual, and finally calculating the optical flow, i.e., the pixel displacement vectors of the two adjacent low-resolution image frames. The optical flow network can be a Spynet network.

[0109] As another possible approach, pixel displacement vectors between two adjacent low-resolution frames in the preparatory low-resolution image can be obtained through deformable convolutional networks.

[0110] Deformable convolutional networks (DCNNs) can add an offset to the sampling position in standard convolution operations. They can include two convolutional kernels: one for generating output features and the other for generating the offset. First, feature maps are extracted using traditional convolutional layers. These feature maps are then input into the convolutional layer (conv) to obtain the offsets between two adjacent low-resolution images, generating a channel dimension of 2N, where 2 corresponds to the offsets in the X and Y dimensions, and N is the number of channels. The two convolutional kernels of the deformable DCNN are simultaneously learned using a bilinear interpolation backpropagation algorithm to obtain the pixel displacement vectors between adjacent low-resolution images.

[0111] First, adjust the position of pixels in the low-resolution images of adjacent frames of the target frame based on the pixel displacement vector. Then, adjust the other low-resolution images in sequence based on the adjusted pixel positions of the low-resolution images of adjacent frames of the target frame, thereby aligning the features of two adjacent low-resolution images.

[0112] S223, extract and fuse the features of the first low-resolution image to obtain the features of the second low-resolution image.

[0113] The second low-resolution image feature is obtained by superimposing features from the pre-low-resolution images after feature alignment. First, the features of consecutive frames of the first low-resolution image are subjected to feature scale transformation, including but not limited to upsampling and downsampling. Then, the feature scale-transformed first low-resolution image features are extracted, and the features of multiple frames of low-resolution images are fused temporally to obtain the second low-resolution image feature. The second low-resolution image feature includes both high-order and low-order features.

[0114] S224, Generate a high-resolution image of the target frame based on the features of the second low-resolution image.

[0115] First, the features of the second low-resolution image are input into a convolutional layer to extract higher-order features, resulting in a higher-order feature map. This higher-order feature map is relatively small in size compared to the features of the second low-resolution image. Then, the higher-order feature map is upsampled to increase its size. Finally, the higher-order feature map is combined with the luminance component of the second low-resolution image features to generate a high-resolution image of the target frame with color.

[0116] S225, stitch together the high-resolution images of the target frame to obtain a true high-resolution bird's-eye view.

[0117] The target frame high-resolution image is a high-resolution image from a single viewpoint. Multiple high-resolution images of the target frame from different viewpoints are stitched together to obtain a ground truth high-resolution bird's-eye view. The ground truth high-resolution bird's-eye view has the same size as the target frame low-resolution bird's-eye view. This ground truth high-resolution bird's-eye view is a true high-resolution bird's-eye view and serves as a reference image for the discriminator in judging the high-resolution bird's-eye view output by the generator.

[0118] The AVM stitching algorithm can be used to stitch together a true high-resolution bird's-eye view to obtain a true high-definition bird's-eye view. This process can include distortion correction, joint calibration, projection transformation, and stitching fusion. The method for stitching a true high-definition bird's-eye view is the same as the stitching method in S520, and the specific implementation process will be described in detail later.

[0119] In one feasible way, such as Figure 4 As shown, S230 may include:

[0120] S231, Based on the low-resolution bird's-eye view of the target frame, generate a predicted high-resolution bird's-eye view using a generator.

[0121] The predicted high-resolution bird's-eye view is a test bird's-eye view continuously generated by the generator based on the input target frame low-resolution bird's-eye view. Its resolution may not reach the resolution of the true high-resolution bird's-eye view at the beginning, but after multiple training sessions, the resolution of the predicted high-resolution bird's-eye view will gradually approach the resolution of the true high-resolution bird's-eye view.

[0122] The low-resolution bird's-eye view of the target frame is input into the generator. The generator extracts the features of the low-resolution bird's-eye view of the target frame and performs upsampling and downsampling to obtain the predicted high-resolution bird's-eye view.

[0123] For every low-resolution bird's-eye view of the target frame input into the generator, the generator will generate a predicted high-resolution bird's-eye view.

[0124] S232, based on the true high-definition bird's-eye view, determine the type information of the predicted high-definition bird's-eye view through a discriminator.

[0125] The type information includes the generator-generated image and the ground truth generated image. The generator-generated image indicates the type of the predicted high-resolution bird's-eye view, while the ground truth generated image indicates a type similar to the ground truth high-resolution bird's-eye view. The discriminator's output can include 0 or 1, where 0 indicates that the predicted high-resolution bird's-eye view is a fake image of type generator-generated image, and 1 indicates that the predicted high-resolution bird's-eye view is a real image of type ground truth generated image.

[0126] The ground truth high-resolution bird's-eye view serves as a reference image and becomes the conditional label for the discriminator. If the discriminator identifies the predicted high-resolution bird's-eye view as extremely similar to or identical to the ground truth high-resolution bird's-eye view, it considers it a ground truth generated image and outputs 1. If the discriminator identifies the predicted high-resolution bird's-eye view as different from the ground truth high-resolution bird's-eye view, it considers it a generator generated image and outputs 0.

[0127] S233, Determine the detailed compensation model based on the type information.

[0128] If the discriminator identifies the type information of the predicted high-resolution bird's-eye view as a ground truth generated image, it indicates that the predicted high-resolution bird's-eye view generated by the generator is already very realistic, and the generator's parameters no longer need to be updated. At this time, the generator can be used as a detail compensation model.

[0129] Furthermore, when the type information is a generator-generated image, a predicted high-resolution bird's-eye view is generated by the generator based on the target low-resolution bird's-eye view.

[0130] The generator is identified as a detail compensation model when the discriminator determines that the type information of the predicted high-definition bird's-eye view is a true value generated image and the recognition rate is greater than a preset threshold.

[0131] When the discriminator identifies the type information of the predicted high-definition bird's-eye view as a generator-generated image, it is still a fake image to the discriminator. The generator updates its internal parameters based on the discriminator's recognition result, so that the output predicted high-definition bird's-eye view gradually approaches the real high-definition bird's-eye view, to the point of being indistinguishable from the real image.

[0132] Whenever the discriminator identifies the type information of the predicted high-resolution bird's-eye view as a generator-generated image, the generator updates its internal parameters and continues to input the target low-resolution bird's-eye view to generate the predicted high-resolution bird's-eye view. Then, it is input into the discriminator to judge the authenticity of the predicted high-resolution bird's-eye view until it is determined that the type information of the predicted high-resolution bird's-eye view is a ground truth generated image. At this point, the image output by the generator is a real image to the discriminator, and the generator is identified as a detail compensation model.

[0133] Furthermore, to enhance the generator's ability to create deceptive images, the generator is designated as a detail compensation model after the discriminator repeatedly classifies the predicted high-resolution bird's-eye view as a ground truth image. Specifically, the recognition rate is defined as the ratio of the number of times the discriminator classifies the predicted high-resolution bird's-eye view as a ground truth image to the total number of discriminations made by the discriminator. When the recognition rate exceeds a preset threshold, it indicates that the predicted high-resolution bird's-eye view has a high probability of causing the discriminator to misjudge, and the generator's generation ability is sufficient to deceive the discriminator. In this case, model training ends, and the generator is designated as a detail compensation model. The preset threshold can be set according to actual needs and is not limited here.

[0134] Figure 5 A flowchart illustrating an image generation method provided in this application embodiment, the method can be performed by, for example... Figure 1 The vehicle-mounted terminal 110 in the system shown executes this. For example... Figure 5 As shown, the method may include the following steps:

[0135] S510, obtain the vehicle's driving environment map.

[0136] Specifically, the vehicle's driving environment map is acquired by the camera device installed in the vehicle. The driving environment map can be an environmental map within a preset distance of the vehicle. The preset distance can be determined according to actual business needs, actual product needs, or actual application scenarios.

[0137] The driving environment map includes a sequence of images from multiple perspectives. Each image in the sequence includes roads, obstacles, and traffic lights. Obstacles are objects that interfere with the driving of vehicles, such as people, vehicles, small animals, fences, etc.

[0138] Multiple perspectives include the vehicle's front, rear, left, and right views. Image sequences from these multiple perspectives can be acquired by placing camera devices at the front, rear, left, and right of the vehicle. For example, four external camera devices can be installed on the vehicle's front bumper, trunk, and rearview mirrors. These camera devices can include image sensors, such as cameras, and the cameras can be multi-channel fisheye cameras, for example, a four-channel fisheye camera.

[0139] During vehicle operation, the camera devices around the vehicle can be activated to capture video images in real time. Image sequences from multiple perspectives can be determined from the captured video images, or the driving environment map captured and saved by the camera devices can be obtained from the vehicle's storage module. Depending on the needs of different application scenarios, the corresponding acquisition method can be selected to obtain the driving environment map.

[0140] The S520 generates low-resolution bird's-eye view images based on a sequence of images from multiple perspectives.

[0141] Each viewpoint contains at least one image sequence, and each image sequence includes at least one low-resolution image frame. A low-resolution bird's-eye view is a three-dimensional representation of the vehicle's driving environment, drawn using high-viewpoint perspective to depict the undulating terrain from a high point. The low-resolution bird's-eye view is obtained by stitching together low-resolution images corresponding to the same frame from multiple viewpoints; therefore, a low-resolution bird's-eye view is a single-frame image. These multiple viewpoints can be, for example, front, rear, left, and right views.

[0142] For example, the low-resolution images of the Tth frame can be selected from the image sequences of the front view, rear view, left view, and right view, and these four low-resolution images can be stitched together to obtain the Tth frame of low-definition bird's-eye view.

[0143] Low-resolution bird's-eye view images have low clarity of details and boundaries, resulting in poor image quality and usability. Therefore, detail compensation processing is required to improve their clarity.

[0144] S530 generates high-resolution bird's-eye views based on low-resolution bird's-eye views using a pre-trained detail compensation model.

[0145] A detail compensation model is a generator that can recover high-frequency detail information, enabling image-to-image conversion. This generator can employ an hourglass-shaped encoder-decoder network structure, such as UNet. The generator includes, but is not limited to, an encoder and a decoder. The encoder performs feature extraction and downsampling on the input low-resolution bird's-eye view. The decoder performs feature fusion and upsampling on the image features obtained from the encoder's processing of the low-resolution bird's-eye view, ultimately producing a high-resolution bird's-eye view. The detail compensation model can be based on... Figure 2 The method shown is used for training.

[0146] Because the generator's input and output structures are roughly aligned, the surface appearances of the input and output images differ, but both have the same structure. This means that the low-resolution and high-resolution bird's-eye view images are the same size and can be directly used for subsequent algorithms. For example, high-resolution bird's-eye view images can be used for algorithms such as parking space detection, drivable area detection, and obstacle detection.

[0147] As can be seen, this embodiment of the application acquires a driving environment map of the vehicle, generates a low-resolution bird's-eye view based on image sequences from multiple perspectives within the driving environment map, and inputs the low-resolution bird's-eye view into a pre-trained detail compensation model to obtain a high-resolution bird's-eye view. Since the detail compensation model can improve the imaging effect and accuracy of the low-resolution bird's-eye view, it provides a highly usable input image for the automatic parking function, increases the utilization rate of the low-resolution bird's-eye view, and acquires more reliable details, which is beneficial for improving the accuracy of any subsequent perception detection algorithm based on bird's-eye view mosaicking as input.

[0148] The steps in the above method flow are described in detail below. First, the above-mentioned S520, namely "generating a low-resolution bird's-eye view based on an image sequence from multiple perspectives," will be described in detail with reference to the embodiment.

[0149] As one possible approach, S520 may include stitching together image sequences from multiple perspectives to obtain a low-resolution bird's-eye view.

[0150] A low-resolution bird's-eye view is obtained by stitching together image sequences from multiple perspectives using the AVM (Aspect-of-Video) stitching algorithm. AVM is a system that uses multiple ultra-wide-angle fisheye lenses to capture images, performs distortion correction on the captured images using a special algorithm, and then stitches them together to form a panoramic image of the surroundings of an object. The special algorithm mentioned above is the AVM stitching algorithm, such as... Figure 6 As shown, the AVM stitching algorithm includes multiple processing steps such as distortion removal, joint calibration, projection transformation, and stitching fusion.

[0151] Specifically, such as Figure 7 As shown, the steps for stitching together image sequences from multiple viewpoints using the AVM stitching algorithm include:

[0152] S521, based on the camera's intrinsic parameters and distortion coefficients of the captured image sequence, performs distortion correction processing on the image sequence from multiple viewpoints to obtain the processed image sequence.

[0153] Among these, a fisheye camera can be used to capture the image sequence. To achieve better image quality, existing cameras typically incorporate a lens in front of the camera. However, the addition of a lens affects light propagation; that is, straight lines in the real world appear as curves in the image. For example, Figure 6 Images captured by the front, rear, left, and right fisheye cameras. To more realistically represent the vehicle's driving environment, distortion correction processing is required on the images captured by the multiple fisheye cameras.

[0154] First, obtain the intrinsic parameters and distortion coefficients of each fisheye camera.

[0155] Specifically, calibration cloth, such as a checkerboard calibration cloth, is placed around the vehicle. Fisheye cameras positioned around the vehicle capture images of its surrounding environment, including portions of the calibration cloth. Figure 6 Images taken from the front-view, rear-view, left-view, and right-view fisheye.

[0156] Based on the calibration cloth in the surrounding environment map, a checkerboard calibration algorithm is used to calculate the intrinsic parameters and distortion coefficients of the vehicle-mounted camera. The coordinates of the intersection points of each adjacent black and white square in the calibration cloth can be transformed into coordinates in the pixel coordinate system. A coordinate transformation is performed between the image coordinate system and the pixel coordinate system, and the first transformation matrix of the pixel coordinates in the world coordinate system is calculated. Simultaneously, the second transformation matrix of the pixel coordinates in the camera coordinate system is also calculated. Based on the first and second transformation matrices, as well as the rotation matrix and translation vector in the world coordinate system, the final matrix of the pixel coordinates in the world coordinate system is calculated. A relatively good initial solution is obtained through the derivation of the final matrix. The optimal solution is obtained through nonlinear optimization, and this optimal solution includes the intrinsic parameters and distortion system.

[0157] Then, based on the intrinsic parameters and distortion coefficients, the image sequences from multiple viewpoints are subjected to distortion correction to obtain the processed image sequences.

[0158] During distortion correction, images corresponding to the same frame in the image sequence captured by the front-view, rear-view, left-view, and right-view fisheyes are processed to obtain a set of processed images. The processed image sequence includes multiple sets of processed images, and each set of processed images includes four processed images.

[0159] S522, perform joint calibration on the processed image sequence based on preset calibration parameters to obtain the projection matrix.

[0160] After distortion correction, the four fisheye cameras are jointly calibrated to ensure that the images acquired by the four fisheye cameras can be stitched together on the ground to form a ring view.

[0161] First, place calibration objects on the ground around the vehicle. Calibration discs or calibration cloths can be used, as long as the calibration plate of the object is clearly visible. Each calibration object should be located exactly within the overlapping area of ​​the fields of view of two adjacent cameras.

[0162] The preset calibration parameters are manually set after the calibration object is placed. These include: inner displacement width, inner displacement height, displacement width, displacement height, total width, total height, and the four corners of the rectangular area where the vehicle is located. Specifically, the inner displacement width is the distance between the inner edge of the calibration object and the left and right sides of the vehicle; the inner displacement height is the distance between the inner edge of the calibration object and the front and rear of the vehicle; the total width is the width of the bird's-eye view; and the total height is the height of the bird's-eye view. The displacement width represents the distance between the height of the calibration object and the bird's-eye view, and the displacement height represents the distance between the width of the calibration object and the bird's-eye view. The preset calibration parameters can be set according to actual needs and are not limited here.

[0163] Then, the distance between the calibration object and the vehicle is adjusted according to the preset calibration parameters. After the calibration object is placed, the projection area of ​​each camera is determined. For example, the projection area corresponding to the front fisheye camera is a rectangle with the inner displacement height, the width of the calibration object, and the sum of the displacement height as the height, and twice the displacement width, twice the width of the calibration object, and the width of the vehicle as the width.

[0164] Finally, based on the calibration objects set up around the vehicle, four fisheye cameras capture images of the calibration objects, and the projection matrix is ​​determined based on the images captured by the four fisheye cameras. The projection matrix is ​​calculated based on the projection transformation relationship between the images. Projection transformation refers to the relationship between multiple images generated by cameras capturing the same plane from different positions and poses. Through projection transformation, the pixel coordinates of the calibration object images can be transformed to the pixel coordinates in the bird's-eye view.

[0165] S523, Perform projection transformation on the processed image sequence according to the projection matrix to obtain at least one initial bird's-eye view.

[0166] The initial bird's-eye view is a preliminary bird's-eye view obtained by projecting a set of processed images from the processed image sequence onto the ground based on a projection matrix. The initial bird's-eye view includes four bird's-eye view images containing overlapping areas.

[0167] The processed image sequence is transformed using a projection matrix, projecting at least one set of processed images onto the plane of a camera taking pictures parallel to the ground above the vehicle, thus obtaining an initial bird's-eye view. This includes one bird's-eye view image each at the front, rear, left, and right of the vehicle, containing overlapping areas.

[0168] S524, stitching and merging at least one initial bird's-eye view to obtain a low-resolution bird's-eye view.

[0169] Because fisheye cameras have overlapping fields of view, there are overlapping areas between the four initial bird's-eye view images. These areas need to be smoothed and blended to make the initial bird's-eye view images look more natural and realistic.

[0170] The smoothing process can employ a weighted averaging method. Specifically, overlapping regions are extracted from the initial bird's-eye view, grayscale and binarized, and morphological operations are used to remove noise, resulting in two black regions and one white region. The white region is then added between the two black regions. The two black regions are triangular, and the distance between the white region pixels and the boundaries of the two black regions is calculated using the hypotenuses as boundaries. A weight is then determined: the closer a white region pixel is to the boundary of a black region, the greater the weight of the corresponding viewpoint image. The viewpoint image with the higher weight in the overlapping region is retained, while the viewpoint image with the lower weight in the overlapping region is removed, thus merging the overlapping parts.

[0171] Multiple bird's-eye view images containing overlapping areas are stitched together from an initial bird's-eye view frame to obtain a low-resolution bird's-eye view frame. Therefore, when a processed image sequence contains multiple sets of processed images, the low-resolution bird's-eye view frame comprises multiple low-resolution bird's-eye view images.

[0172] The following describes in detail S530, namely, "Generate a high-resolution bird's-eye view based on a pre-trained detail compensation model from a low-resolution bird's-eye view," with reference to the embodiments.

[0173] As a feasible approach, such as Figure 8 As shown, S530 may include:

[0174] S531 generates a first feature map based on a low-resolution bird's-eye view using an encoder.

[0175] The encoder includes a first convolutional layer and a downsampling layer, and the first feature map can include low-order features and high-order features. A low-resolution bird's-eye view is input into the detail compensation model, and the encoder in the detail compensation model performs feature extraction and downsampling on the low-resolution bird's-eye view to obtain the first feature map.

[0176] Specifically, an initial feature map is generated through the first convolutional layer based on a low-resolution bird's-eye view;

[0177] Based on the initial feature map, a first feature map is generated through a downsampling layer.

[0178] First, after the low-resolution bird's-eye view is input into the detail compensation model, features are extracted through the first convolutional layer of the encoder to obtain an initial feature map. This initial feature map can include both low-order and high-order features. Next, the initial feature map is input into a downsampling layer to obtain a smaller first feature map, making it a lower-dimensional representation that can extract as many low-order and high-order features as possible.

[0179] In low-resolution bird's-eye view images, the shallower the convolutional layers, the more detailed the first feature map, recording information such as color and texture; deeper layers are more abstract, recording higher-level information. Based on this, the encoder can include multiple sets of first convolutional layers and downsampling layers. The encoder structure can be: first convolutional layer - downsampling layer - first convolutional layer - downsampling layer - first convolutional layer - downsampling layer... Due to the high flexibility and learnability of neural networks, the combination of first convolutional layers and downsampling layers is not fixed and can be set according to actual needs. The size of the low-resolution bird's-eye view image decreases as the number of downsampling layers increases.

[0180] S532, based on the first feature map, generates a high-definition bird's-eye view through the decoder.

[0181] The decoder includes a second convolutional layer and an upsampling layer. A low-resolution bird's-eye view is input into the detail compensation model. The decoder in the detail compensation model extracts low-order and high-order features from the first feature map, combines them, and then upsamples the combined features to reconstruct a high-resolution bird's-eye view. The low-resolution bird's-eye view is the same size as the high-resolution bird's-eye view.

[0182] Specifically, a combined feature map is generated through a second convolutional layer based on the first feature map;

[0183] A high-resolution bird's-eye view is generated by upsampling the combined feature map.

[0184] First, after the encoder generates the first feature map, the low-order and high-order features in the first feature map are extracted and combined through a second convolutional layer to obtain a combined feature map, which includes both low-order and high-order features. Next, the combined feature map is input into an upsampling layer to gradually restore its size, resulting in a high-resolution bird's-eye view that is the same size as the low-resolution bird's-eye view.

[0185] It should be noted that the decoder can include multiple sets of second convolutional layers and upsampling layers. The decoder structure can be second convolutional layer-upsampling layer-second convolutional layer-upsampling layer-second convolutional layer-upsampling layer... The number of sets of second convolutional layers and upsampling layers in the decoder can be the same as the number of sets of first convolutional layers and downsampling layers in the encoder, or they can be different, as long as the total upsampling factor of the decoder and the total downsampling factor of the encoder are the same, thereby ensuring that the size of the low-resolution bird's-eye view is the same as the size of the high-resolution bird's-eye view.

[0186] As an feasible approach, the detail compensation model is based on training an adversarial generative network.

[0187] Generative Adversarial Networks (GANs) are deep learning models consisting of a generator and a discriminator. The generator produces a new high-resolution bird's-eye view based on an input low-resolution bird's-eye view. The discriminator judges whether the new high-resolution bird's-eye view is real or fake based on a given real bird's-eye view. If the discriminator determines the new high-resolution bird's-eye view is fake, the generator continues to produce a new high-resolution bird's-eye view based on the input low-resolution view, striving to generate a realistic bird's-eye view to deceive the discriminator. The discriminator then tries to distinguish the generator's low-resolution view from the real bird's-eye view. Through this game-like interaction between the generator and discriminator, the generator learns to produce relatively good outputs until it can generate fairly realistic bird's-eye views. At this point, training of the GAN stops, and the generator is used as a detail compensation model.

[0188] It should be understood that, although Figures 2-5 , Figure 7 , Figure 8 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this application, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Furthermore, Figures 2-5 , Figure 7 , Figure 8 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0189] According to embodiments of this application, this application also provides a computer device and a computer-readable storage medium.

[0190] like Figure 9 The diagram shown is a block diagram of a computer device according to an embodiment of this application. The term "computer device" is intended to represent various forms of digital computers or mobile devices. The digital computer may include a desktop computer, a portable computer, a workbench, a personal digital assistant, a server, a mainframe computer, and other suitable computers. The mobile device may include a tablet computer, a smartphone, a wearable device, etc.

[0191] like Figure 9 As shown, device 900 includes a computing unit 901, ROM 902, RAM 903, bus 904, and input / output (I / O) interface 905. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0192] The computing unit 901 can execute various processes in the method embodiments of this application according to computer instructions stored in the read-only memory (ROM) 902 or computer instructions loaded from the storage unit 908 into the random access memory (RAM) 903. The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. The computing unit 901 can include, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. In some embodiments, the methods provided in the embodiments of this application can be implemented as computer software programs, which are tangibly contained in a computer-readable storage medium, such as the storage unit 908.

[0193] RAM 903 can also store various programs and data required for the operation of device 900. Part or all of the computer program can be loaded and / or installed on device 900 via ROM 802 and / or communication unit 909.

[0194] The input unit 906, output unit 907, storage unit 908, and communication unit 909 in device 900 can be connected to I / O interface 905. The input unit 906 can be, for example, a keyboard, mouse, touchscreen, or microphone; the output unit 907 can be, for example, a display, speaker, or indicator light. Device 900 can exchange information and data with other devices through the communication unit 909.

[0195] It should be noted that the device may also include other components necessary for normal operation. It may also include only the components necessary for implementing the solution of this application, without necessarily including all the components shown in the figures.

[0196] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), payload programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof.

[0197] The computer instructions used to implement the methods of this application may be written in any combination of one or more programming languages. These computer instructions may be provided to the computing unit 901 such that when executed by the computing unit 901, such as a processor, the computer instructions cause the execution of the steps involved in the embodiments of the methods of this application.

[0198] The computer-readable storage medium provided in this application can be a tangible medium that can contain or store computer instructions for performing the steps involved in the method embodiments of this application. The computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, and other forms of storage media.

[0199] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method includes: Acquire a sequence of low-resolution images captured by a vehicle camera from multiple perspectives, the sequence of low-resolution images comprising multiple frames of low-resolution images; Based on the multiple low-resolution images, generate a low-resolution bird's-eye view of the target frame and a true high-resolution bird's-eye view; Based on the multiple low-resolution images, a ground truth high-resolution bird's-eye view is generated, including: Based on the position of the target frame low-resolution image in the multiple low-resolution images, determine the preparation low-resolution image; Align the image features of two adjacent low-resolution images in the prepared low-resolution image to obtain the first low-resolution image feature; Extract and fuse the features of the first low-resolution image to obtain the features of the second low-resolution image; Based on the features of the second low-resolution image, a high-resolution image of the target frame is generated; By stitching together the high-resolution images of the target frame, a true high-resolution bird's-eye view is obtained; An adversarial generative network is trained based on the low-resolution bird's-eye view of the target frame and the ground truth high-resolution bird's-eye view to obtain a detail compensation model.

2. The method according to claim 1, characterized in that, The step of generating a low-resolution bird's-eye view of the target frame based on the multiple low-resolution images includes: By stitching together the target low-resolution image from the multiple low-resolution images, a low-resolution bird's-eye view of the target frame is obtained. The target low-resolution image is a low-resolution image located in the same frame from multiple low-resolution images from different perspectives.

3. The method according to claim 1, characterized in that, The step of determining the preliminary low-resolution image based on the position of the target frame low-resolution image in the multiple low-resolution images includes: Based on the position of the target frame low-resolution image in the multiple low-resolution images, the preparation low-resolution image is determined based on a sliding window of a preset size, wherein the target frame low-resolution image is located in the middle of the sliding window.

4. The method according to claim 1, characterized in that, The step of aligning the image features of two adjacent low-resolution images in the prepared low-resolution image to obtain the first low-resolution image features includes: Obtain the pixel displacement vectors of two adjacent low-resolution frames in the prepared low-resolution image; Based on the pixel displacement vector, the image features of two adjacent low-resolution images in the prepared low-resolution image are aligned to obtain the first low-resolution image feature.

5. The method according to claim 4, characterized in that, The step of obtaining the pixel displacement vectors of two adjacent low-resolution frames in the prepared low-resolution image includes: The pixel displacement vectors of two adjacent low-resolution images in the prepared low-resolution image are obtained by using an optical flow estimation algorithm.

6. The method according to claim 4, characterized in that, The step of obtaining the pixel displacement vectors of two adjacent low-resolution frames in the prepared low-resolution image includes: The pixel displacement vectors of two adjacent low-resolution images in the prepared low-resolution image are obtained through a deformable convolutional network.

7. The method according to claim 1, characterized in that, The adversarial generative network includes a generator and a discriminator. The adversarial generative network is trained based on the low-resolution bird's-eye view of the target frame and the ground truth high-resolution bird's-eye view to obtain a detail compensation model, including: Based on the low-resolution bird's-eye view of the target frame, a predicted high-resolution bird's-eye view is generated by the generator. Based on the true high-resolution bird's-eye view, the discriminator determines the type information of the predicted high-resolution bird's-eye view; Based on the type information, determine the detailed compensation model.

8. The method according to claim 7, characterized in that, The type information includes a generator generation graph and a truth generation graph. Determining the detail compensation model based on the type information includes: When the type information is a generator-generated image, a predicted high-resolution bird's-eye view is generated by the generator based on the target frame low-resolution bird's-eye view. The generator is determined to be a detail compensation model when the discriminator determines that the type information of the predicted high-definition bird's-eye view is a ground truth generated image and the recognition rate is greater than a preset threshold.

9. An image generation method, characterized in that, The method includes: Acquire a driving environment map of the vehicle, the driving environment map including image sequences from multiple perspectives; Based on the image sequence from the multiple viewpoints, a low-resolution bird's-eye view is generated, including: performing distortion correction processing on the image sequence from the multiple viewpoints based on the intrinsic parameters and distortion coefficients of the camera that captured the image sequence, to obtain a processed image sequence; The processed image sequence is jointly calibrated based on preset calibration parameters to obtain a projection matrix; The processed image sequence is projected and transformed according to the projection matrix to obtain at least one initial bird's-eye view; By stitching and fusing the at least one initial bird's-eye view, a low-resolution bird's-eye view is obtained; A high-resolution bird's-eye view is generated based on the low-resolution bird's-eye view and a pre-trained detail compensation model, wherein the detail compensation model is trained by the model training method as described in any one of claims 1 to 8.

10. The method according to claim 9, characterized in that, The detail compensation model includes an encoder and a decoder. The step of generating a high-resolution bird's-eye view based on the low-resolution bird's-eye view and a pre-trained detail compensation model includes: Based on the low-resolution bird's-eye view, a first feature map is generated by the encoder; A high-resolution bird's-eye view is generated by the decoder based on the first feature map.

11. The method according to claim 10, characterized in that, The encoder includes a first convolutional layer and a downsampling layer. The step of generating a first feature map based on the low-resolution bird's-eye view using the encoder includes: Based on the low-resolution bird's-eye view, an initial feature map is generated through the first convolutional layer; Based on the initial feature map, a first feature map is generated through the downsampling layer.

12. The method according to claim 10, characterized in that, The decoder includes a second convolutional layer and an upsampling layer. The step of generating a high-resolution bird's-eye view based on the first feature map using the decoder includes: Based on the first feature map, a combined feature map is generated through the second convolutional layer; A high-resolution bird's-eye view is generated through the upsampling layer based on the combined feature map.

13. A computer device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores computer instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 9-12.

14. A computer-readable storage medium storing computer instructions thereon, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 9-12.

Images