Multi-view image generation method and device, electronic equipment and readable storage medium

Abstract

The invention provides a multi-view image generation method and device, electronic equipment and a readable storage medium, and relates to the technical field of computers. The method comprises the steps of determining a target text; for each initial image in the plurality of initial images, de-noising the initial image by using a target generation sub-model in a trained target diffusion model based on the initial image and a target text to obtain a target image corresponding to the initial image, the target diffusion model being obtained by training based on a plurality of sample sets, each sample set comprises a plurality of sample images obtained by carrying out image acquisition on the same scene from different visual angles, overlapping region features corresponding to the plurality of sample images, and sample texts; the overlapped region feature of each sample image is an image feature determined based on a local region image overlapped with the view of the sample image in the corresponding reference sample image, and the reference sample image and the sample image are located in the same sample set. Therefore, multi-view consistent image data can be generated.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multi-view image generation method, device, electronic device and readable storage medium. Background Art

[0002] Autonomous driving technology is an important development direction in the field of artificial intelligence and computer vision in recent years. Autonomous driving systems rely on a large amount of image training data, which usually includes multi-perspective information of roads, vehicles, pedestrians and other traffic elements in different scenarios and conditions in the real world. However, there are challenges in obtaining this data in the real world. For example, data collection is costly and usually requires specialized equipment and personnel, especially when collecting diverse scenarios (such as bad weather, different geographical locations, complex traffic conditions, etc.). Therefore, how to efficiently obtain multi-perspective autonomous driving image training data has become a bottleneck problem in the current development of autonomous driving technology. Summary of the invention

[0003] The embodiments of the present application provide a multi-perspective image generation method, device, electronic device and readable storage medium, which can generate multi-perspective consistent image data.

[0004] The embodiments of the present application can be implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a multi-view image generation method, the method comprising:

[0006] Determining a target text, wherein the target text is used to indicate the image content of the target image;

[0007] For each initial image among the multiple initial images, a target generation sub-model in a trained target diffusion model is used to perform denoising processing on the initial image based on the initial image and the target text to obtain a target image corresponding to the initial image, wherein the target diffusion model is trained based on multiple sample sets, each of the sample sets includes multiple sample images, overlapping area features corresponding to the multiple sample images, and sample texts, the multiple sample images in a sample set are multiple images obtained by capturing images of the same scene from different perspectives, the overlapping area features of each sample image are image features determined based on a local area image in a corresponding reference sample image that overlaps with the field of view of the sample image, and the reference sample image and the sample image are in the same sample set.

[0008] In a second aspect, an embodiment of the present application provides a multi-view image generation device, the device comprising:

[0009] A text determination module, used to determine a target text, wherein the target text is used to indicate the image content of a target image;

[0010] An image generation module is used to perform denoising processing on each initial image among multiple initial images based on the initial image and target text using a target generation sub-model in a trained target diffusion model to obtain a target image corresponding to the initial image, wherein the target diffusion model is trained based on multiple sample sets, each of the sample sets includes multiple sample images, overlapping area features corresponding to the multiple sample images, and sample text, the multiple sample images in a sample set are multiple images obtained by capturing images of the same scene from different perspectives, the overlapping area features of each sample image are image features determined based on a local area image in a corresponding reference sample image that overlaps with the field of view of the sample image, and the reference sample image and the sample image are in the same sample set.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the multi-view image generation method described in the aforementioned embodiment.

[0012] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multi-view image generation method as described in the aforementioned embodiment is implemented.

[0013] The multi-view image generation method, device, electronic device and readable storage medium provided by the embodiment of the present application first obtain a target text for indicating the image content of the target image; then, for each of the multiple initial images, the target generation sub-model in the trained target diffusion model is used to perform denoising on the initial image based on the initial image and the target text, so as to obtain a target image under a viewing angle corresponding to the initial image. The target diffusion model is trained based on multiple sample sets, each of which includes multiple sample images, overlapping area features corresponding to each of the multiple sample images, and sample text. The multiple sample images in a sample set are multiple images obtained by image acquisition of the same scene from different viewing angles. The overlapping area features of each sample image are image features determined based on the local area images in the corresponding reference sample image that overlap with the field of view of the sample image, and the reference sample image and the sample image are in the same sample set. In this way, by constraining the overlapping area features of multiple viewing angles in the process of training the target diffusion model, the target generation sub-model in the target diffusion model is used to generate image data consistent with multiple viewing angles, thereby reducing the error caused by scene inconsistency, and improving the performance and reliability of the autonomous driving model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application;

[0016] Figure 2 One of the flowcharts of the multi-view image generation method provided in the embodiment of the present application;

[0017] Figure 3 A second flowchart of a multi-view image generation method provided in an embodiment of the present application;

[0018] Figure 4 This is a schematic diagram of the camera visible area of ​​the unmanned vehicle;

[0019] Figure 5 for Figure 3 A schematic flow chart of the sub-steps included in step S130;

[0020] Figure 6 A Unet noise prediction framework diagram provided in an embodiment of the present application;

[0021] Figure 7 One of the block diagrams of the multi-view image generation device provided in an embodiment of the present application;

[0022] Figure 8 The second block diagram of the multi-view image generation device provided in an embodiment of the present application.

[0023] Icons: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication unit; 200 - multi-view image generation device; 210 - training module; 220 - text determination module; 230 - image generation module. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0026] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0027] To overcome the problem of insufficient image training data for autonomous driving, researchers rely on virtual data generation technology to supplement training data with computer-generated images and videos. These virtual data can augment the dataset by simulating different viewpoints, lighting, weather, and traffic conditions. However, there is a significant problem with current virtual viewpoint generation technology, that is, the consistency of the scene cannot be fully guaranteed when generating multiple viewpoints. In practical applications, autonomous driving systems need to be able to process data from different viewpoints from multiple sensors, such as front cameras, side cameras, and radar sensors. The data between different viewpoints needs to be consistent to ensure that the autonomous driving model can correctly understand and infer the objects and dynamics in the scene. However, existing virtual viewpoint generation methods often generate images from each viewpoint independently, resulting in differences in the position, size, color, etc. of the objects in the generated scenes, which leads to the introduction of noise and errors when training autonomous driving models. This scene inconsistency makes it difficult for autonomous driving models to accurately understand and predict dynamic changes in the scene, reducing the performance and reliability of the model.

[0028] Existing methods mainly generate virtual perspectives through simple image transformation and splicing. These methods cannot guarantee the consistency of scenes under multiple perspectives. For example, although traditional 3D rendering technology can generate images from different perspectives, it cannot guarantee consistency, and still faces problems such as insufficient accuracy and high consumption of computing resources in complex real-world scene simulations. In addition, these 3D methods cannot dynamically adjust the correlation between perspectives, which may introduce noise and errors during model training.

[0029] In response to the above situation, the embodiments of the present application provide a multi-perspective image generation method, device, electronic device and readable storage medium. By constraining the overlapping area features of multiple perspectives in the process of training a target diffusion model, the target generation sub-model in the target diffusion model is used to generate multi-perspective consistent image data, thereby reducing errors caused by scene inconsistency and improving the performance and reliability of the autonomous driving model.

[0030] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] Please refer to Figure 1 , Figure 1 A block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be, but is not limited to, a computer, a server, etc. The electronic device 100 includes a memory 110, a processor 120, and a communication unit 130. The memory 110, the processor 120, and the communication unit 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0032] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0033] The processor 120 is used to read / write data or programs stored in the memory 110 and perform corresponding functions. For example, the memory 110 stores a multi-view image generating device 200, and the multi-view image generating device 200 includes at least one software function module that can be stored in the memory 110 in the form of software or firmware. The processor 120 executes various functional applications and data processing by running software programs and modules stored in the memory 110, such as the multi-view image generating device 200 in the embodiment of the present application, that is, realizing the multi-view image generating method in the embodiment of the present application.

[0034] The communication unit 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through a network, and to send and receive data through the network.

[0035] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0036] Please refer to Figure 2 , Figure 2 This is one of the flow diagrams of the multi-view image generation method provided in the embodiment of the present application. The method can be applied to the above-mentioned electronic device. The specific flow of the multi-view image generation method is described in detail below. In this embodiment, the method may include steps S140 to S150.

[0037] Step S140, determining the target text.

[0038] Step S150, for each initial image among the multiple initial images, using the target generation sub-model in the trained target diffusion model based on the initial image and the target text, denoising the initial image to obtain a target image corresponding to the initial image.

[0039] In this embodiment, the electronic device may store a pre-trained target diffusion model. The target diffusion model is obtained by training based on multiple sample sets. A sample set includes multiple sample images, overlapping area features corresponding to each of the multiple sample images, and sample texts corresponding to each of the multiple sample images. The multiple sample images in a sample set are multiple images obtained by capturing images of the same scene from different perspectives. The scenes corresponding to different sample sets may be different. The overlapping area features of each sample image are image features determined based on the local area image in the corresponding reference sample image that overlaps with the field of view of the sample image, and the reference sample image and the sample image are in the same sample set. For example, a sample set includes sample image 1 and sample image 2, and the field of view corresponding to sample image 1 overlaps with the field of view corresponding to sample image 2. Then, the real space area corresponding to the overlapping field of view has partial images in sample image 1 and sample image 2, and the overlapping area features corresponding to sample image 1 can be determined based on the partial image of sample image 2 (i.e., the above-mentioned partial image).

[0040] When it is necessary to generate a multi-view image, a target text may be determined first. The target text is used to indicate the image content of the generated multi-view image, that is, to indicate the image content of multiple target images to be generated. The specific content of the target text may be determined in combination with actual needs, for example, to indicate the generation of multiple target images corresponding to rainy days.

[0041] The multiple initial images may be determined in any manner, and the number of the initial images is the same as the number of multi-view images to be generated, that is, the same as the number of viewing angles. The multiple initial images may be the same image or different images.

[0042] For each initial image, the target generation submodel in the target diffusion model can be used to perform denoising on the initial image based on the initial image and the target image, thereby obtaining a target image corresponding to the initial image. In this way, the target generation submodel can be used to obtain multiple target images based on the target text and multiple initial images, and the multiple target images meet the scene consistency requirements; and the association between perspectives can be dynamically adjusted through the target text.

[0043] In this embodiment, by constraining the overlapping area features of multiple perspectives during the process of training the target diffusion model, the target generation sub-model in the target diffusion model is used to generate multi-perspective consistent image data, thereby reducing the errors caused by scene inconsistency and improving the performance and reliability of the autonomous driving model.

[0044] The following first introduces how to obtain the target diffusion model.

[0045] Please refer to Figure 3 , Figure 3 The second flowchart of the multi-view image generation method provided in the embodiment of the present application is as follows. In this embodiment, before step S140, the method may further include steps S110 to S130.

[0046] Step S110, obtaining a plurality of sample image groups and sample texts corresponding to each sample image in the plurality of sample image groups.

[0047] Step S120 : for each sample image, according to the sample image group to which the sample image belongs, obtaining the overlapping region feature corresponding to the sample image.

[0048] Step S130: training the target diffusion model according to the obtained multiple sample sets.

[0049] In actual environments, there is a certain degree of overlap in the fields of view between the multiple cameras used to obtain multi-view images. Since the relative positions between the cameras remain unchanged, the overlapping area will also remain unchanged. This feature can be used to constrain the perspective consistency.

[0050] For example, in Figure 4 In the schematic diagram of the visible area of ​​multiple cameras of the unmanned vehicle shown, the field of view of camera No. 1 overlaps with that of camera No. 2 and camera No. 3 to a certain extent, and since the relative positions of the cameras remain unchanged, the overlapping area will also remain unchanged. A feature can be used to constrain the consistency of the viewing angle.

[0051] Multiple sample image groups may be acquired. A sample image group includes sample images of the same scene under multiple viewing angles. Optionally, as a possible implementation, Figure 4 As shown, the unmanned vehicle is actually equipped with 6 cameras, and the images obtained simultaneously by the 6 cameras can be used as a frame of data. The frame of data includes an image taken by each of the 6 cameras, and one frame of data is used as a sample image group.

[0052] For a sample image group, the sample text corresponding to each sample image in the sample image group can be obtained. Optionally, the sample text corresponding to each sample image can be obtained by manual annotation.

[0053] Alternatively, a pre-trained first image encoder can be used to encode a sample image (i.e., feature extraction) to obtain a first sample image feature corresponding to the sample image; then, a text generation model can be used to obtain a sample text corresponding to the sample image based on the first sample image feature. By repeating the above process on each sample image obtained, the sample text corresponding to each sample image can be obtained. Among them, the first image encoder can be, but not limited to, a CLIP (Contrastive Language-Image Pretraining) model, a BERT model (Bidirectional Encoder Representations from Transformers), etc. The text generation model can be, but not limited to, a GPT-3 model, etc. In this way, the first image encoder and the text generation model can be used to generate a description matching the image content of the sample image as the sample text corresponding to the sample image through the mapping between image features and text features.

[0054] By performing the above processing on each sample image obtained from each sample image group, a sample image corresponding to each sample image can be obtained, that is, an image-text pair is obtained.

[0055] Each sample image group includes multiple sample images, and the following processing can be performed for each sample image group: for each sample image in the sample image group, determine the reference sample image corresponding to the sample image in the sample image group, the reference sample image and the sample image are in the same sample set and have overlapping fields of view; then, for each reference sample image, determine from the reference sample image a local image corresponding to the overlapping portion of the field of view of the sample image, that is, the local image is the local image in the reference sample image; then, map the local image to the coordinate system used by the sample image to obtain the target local image; finally, obtain the overlapping area features corresponding to the sample image according to the target local image. The overlapping area features are obtained after encoding the target local image (that is, feature extraction).

[0056] For example, Figure 4 In the scene shown, there are 6 images in one frame of data (i.e., a sample image group includes 6 sample images). The mapping relationship between images with overlapping areas can be calculated first. In many autonomous driving datasets (such as Nuscenes), a view usually overlaps with its left and right views. Therefore, for the i-th view (i.e., the i-th sample image in a sample image group), only the left view i l and right view i r . Available from view i r The mapping relationship between the overlapping area of ​​​​the view i and the mapping function is represented by g(*). r The a coordinate in the left view i is mapped to the b coordinate in the left view i (the calculation process is b = g(a)). l and right view i r The local images in the image are mapped to the corresponding overlapping fields of view i, and then the local images are mapped in combination with the mapping relationship to obtain two target local images in the coordinate system used by view i, and then two sets of overlapping area features are obtained based on the two target local images, one of which corresponds to view i. l , another set of overlapping region features corresponding to view i r .

[0057] Through the above processing, multiple sample sets can be obtained. Among them, one sample set includes multiple samples, and one sample can include a sample image, the overlapping area features corresponding to the sample image, and the sample text corresponding to the sample image. The sample images of a sample set are a sample image group.

[0058] When multiple sample sets are available, Figure 5 The initial diffusion model is trained in the manner shown to obtain the target diffusion model. Figure 5 , Figure 5 for Figure 3 Schematic diagram of the flow of sub-steps included in step S130. In this embodiment, step S130 may include sub-steps S131 to S135.

[0059] Sub-step S131, for each sample in the sample set, adding noise to the sample image in the sample through the initial noisy network to obtain a noisy feature map.

[0060] Sub-step S132, using the initial denoising network to denoise the noisy feature map based on the sample text and overlapping area features corresponding to the sample image, and obtain the noise loss corresponding to the sample image.

[0061] Sub-step S133, obtaining a total noise loss corresponding to the sample set according to the noise loss corresponding to each sample in the sample set.

[0062] Sub-step S134, adjusting parameters of an initial diffusion model according to the total noise loss, wherein the initial diffusion model includes an initial denoising network and an initial denoising network.

[0063] Sub-step S135, when it is determined that the training stop condition is not met, continue the training; when it is determined that the training stop condition is met, stop the training to obtain the target diffusion model.

[0064] In this embodiment, the initial diffusion model may include an initial denoising network and an initial denoising network. During the training process, for each sample set, the initial loading network may be used to perform denoising on each sample image in the sample set to obtain a denoised feature map, and then the initial denoising network may be used to perform denoising on each obtained denoised feature map, so that the denoising and denoising of the sample image can be completed. Also, for the denoising and denoising of each sample image, the noise loss of the sample image is calculated, and then the total noise loss corresponding to the sample set is calculated, and then the parameters of the initial diffusion model may be adjusted according to the total noise loss corresponding to the obtained sample set. Also, it is determined whether the training stop condition is met. If so, the current initial diffusion model may be used as the trained target diffusion model. If the training stop condition is not met, the above process may be repeated based on the sample set until the training stop condition is met.

[0065] Assuming that the random time step corresponding to a sample set is t, the Gaussian noise used in the noise addition process of each sample image in the sample set corresponds to the random time step t. The function of the initial denoising network is to predict the feature z_t-1 before noise addition, so as to determine the noise between t-1 and t (i.e., estimated noise) based on the feature z_t-1 before noise addition and the feature z_t after noise addition, and then calculate the noise loss based on the noise and the noise determined based on the random time step t.

[0066] The initial denoising network can obtain the output result of the cross attention according to the sample text and overlapping area features corresponding to the sample image, and then perform denoising based on the output result. In this way, the target model can better understand the relationship between the image to be generated, the text, and the overlapping area.

[0067] The output result of the cross attention is obtained based on the Query matrix, the Key matrix and the Value matrix. The Query matrix is ​​obtained based on the sample image, and the Key matrix and the Value matrix are obtained based on the sample text and overlapping area features corresponding to the sample image.

[0068] The total noise loss can be obtained by calculating the mean square error loss, and the calculation formula can be shown as follows: Among them, E is the expected value, which represents the average value of the noise loss corresponding to the sample image in a sample set; x represents the input original data sample (i.e., sample image); ∈ represents the standard normal distribution noise, X t represents the noisy image at time step t; ∈ θ (*) represents the noise predicted by the model; the parameter θ represents the weight of the model (i.e., the parameter of the model); T represents the control condition of the sample text, and C represents the control condition of the overlapping area feature.

[0069] Optionally, the training stop condition may be determined to be satisfied when the total noise loss is less than a preset value or the number of iterations of the model reaches a preset number. The training stop condition may also be set in combination with actual needs, which is not specifically limited here.

[0070] For example, after the number of training iterations, if the loss is observed to continue to decrease and stabilize, samples can be generated regularly and quality assessment can be performed to observe whether the visual effect is stable. If the loss function value is less than the threshold and the generation effect is good, the model parameters are determined to be the parameters of the target diffusion model. If it is greater than the threshold, it indicates that the training is not good and the learning rate and early stopping strategy need to be adjusted to continue training.

[0071] In this embodiment, the initial diffusion model may further include a second image encoder, an image decoder and a text encoder. The second image encoder may be used to encode the sample image (i.e., obtain the second sample image feature), and then perform noise processing, and then the data obtained after the noise processing is input into the initial denoising network. The text encoder may also be used to encode the sample text, and the text encoding result and the corresponding overlapping area feature are also input into the initial denoising network, and the initial denoising network will denoise the obtained data, and the final denoising result is input into the image decoder for decoding to obtain a generated image corresponding to the input sample image. Among them, the image decoder may be, but not limited to, a VAE decoder.

[0072] As a possible implementation method, the initial denoising network is a Unet network (also called a Unet denoising network). Figure 6 The above training process is briefly illustrated with examples.

[0073] like Figure 6 As shown, the initial diffusion model may include a second image encoder (i.e. Figure 6 The encoder in the image decoder (i.e., the initial noise adding network, the Unet network, and the image decoder (i.e., Figure 6 The decoder in ) and the text encoder (i.e. Figure 6 Encoder corresponding to Chinese text).

[0074] For each sample image in a sample set, the text features corresponding to the sample image (i.e., the features obtained after the sample text is processed by the text encoder) and the overlapping area features are used as control conditions and input into the Unet denoising network. Figure 6 In the latent feature space, we need to select a random time step t for the currently used sample set, apply the Gaussian noise corresponding to the random time step t to the image feature z_0 corresponding to each sample image, obtain the noisy feature z_t, and convert the time step t into a vector.

[0075] The Unet denoising network is used to predict the feature z_t-1 before adding noise. In the above description, the text features corresponding to the sample text and the overlapping area features are input into the Unet denoising network as control conditions. After that, the cross attention mechanism of Transformer can be applied to enable the target diffusion model to better understand the relationship between the image to be generated, the text, and the overlapping area. Assume that the image feature of the current sample image is X I , the corresponding text feature of the sample text is X T , the overlapping area feature is X C , the cross attention calculation formula is as follows:

[0076]

[0077] Where Q represents the Query matrix, K represents the Key matrix, and V represents the Value matrix; W Q , W K ,W,represent the learned weight matrices, which are used to generate Q, K, and V respectively.

[0078] The output of the cross attention is calculated by the following formula:

[0079]

[0080] Among them, QK T It is the dot product of the transposed matrix of the Query matrix and the Key matrix, and the result is an attention score matrix that represents the correlation between each position in the input sequence. is a trick scaling factor, usually taken as the dimension d of the Key matrix k The square root of is used to avoid excessive dot product values ​​when the dimension is large, which affects the gradient and calculation of Softmax. The Softmax function is used to convert attention scores into weights.

[0081] In this way, text information and the overlapping area information between the two cameras are input to the model to achieve cross-view consistency. The text, image, and overlapping area are used to capture the correlation between different parts of the input data through a cross-attention mechanism. This is done by generating Q, K, and V through linear transformation, and then calculating the attention score and weighted sum, which enhances the model's ability to capture multiple features and associations.

[0082] For each sample image in a sample set, the noisy feature z_t corresponding to the sample image can be obtained by continuously removing Gaussian noise through the Unet denoising network to obtain the image potential code z_0, and then the generated image can be obtained through the second image decoder.

[0083] After completing the denoising and denoising process of a sample image, the noise loss corresponding to each sample image can be calculated. Afterwards, when the noise loss corresponding to multiple sample images in a sample set is obtained, the average value of the noise loss corresponding to the multiple sample images can be used as the total noise loss corresponding to the sample set. Afterwards, the total noise loss corresponding to the multiple sample sets can be obtained, and the parameters of the initial diffusion model can be adjusted to obtain the target diffusion model.

[0084] In the training process, when denoising a sample image, the camera parameter information corresponding to the sample image can also be input into the Unet denoising network, and the Unet denoising network also performs denoising in combination with the camera parameter information. The camera parameter information can be used to describe the information of the camera that obtains the sample image, such as the external parameters, internal parameters, and position of the camera. In a sample set, different sample images correspond to different camera parameter information.

[0085] The target generation sub-model in the target diffusion model may include a target image encoder, a target image decoder, a target text encoder and a target denoising network. The target image encoder is the second image encoder in the initial diffusion model when training is stopped, the target image decoder is the image decoder in the initial diffusion model when training is stopped, the target text encoder is the text encoder in the initial diffusion model when training is stopped, and the target denoising network is the initial denoising network in the initial diffusion model when training is stopped.

[0086] In this case, for each initial image among the multiple initial images, the initial image can be encoded by the target image encoder to obtain initial image features; and the target text can be encoded by the target text encoder to obtain text features; then, the initial image features are denoised by the target denoising network according to the initial image features and text features to obtain denoised initial image features; finally, the target image decoder is used to decode the image feature map after denoising the initial image features to obtain the target image.

[0087] In the case where camera parameters are used during the training process, correspondingly, the target image under the viewing angle corresponding to the camera parameter information can be obtained through the target generation sub-model based on the camera parameter information corresponding to the initial image, the initial image and the target text. Among them, the camera parameter information corresponding to each of the multiple initial images is different. When the target generation sub-model includes a target image encoder, a target image decoder, a target text encoder and a target denoising network as described above, the camera parameter information corresponding to the initial image can be encoded, and then input into the target denoising network together with the initial image features and text features corresponding to the initial image, and then the final output of the target denoising network is decoded by the target image decoder to obtain the target image corresponding to the initial image, and the target image corresponds to the camera parameter information corresponding to the initial image.

[0088] In order to maintain scene consistency when generating images from multiple virtual perspectives, in this embodiment, the model training is performed by constraining the overlapping area features of multiple perspectives to obtain a target diffusion model, and then the target generation sub-model in the target diffusion model is used to generate images with multiple perspectives that are consistent. The multi-perspective images generated in this way are used as training data for the autonomous driving model, which can reduce errors caused by scene inconsistency and improve the performance and reliability of the autonomous driving model. The above method can be used as a method for generating image data for unmanned vehicles with scene consistency to obtain multi-perspective image data for training unmanned driving models.

[0089] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a multi-view image generating device 200 is given below. Optionally, the multi-view image generating device 200 can adopt the above Figure 1 The device structure of the electronic device 100 is shown. Figure 7 , Figure 7 This is one of the block diagrams of the multi-view image generation device 200 provided in the embodiment of the present application. It should be noted that the basic principle and technical effects of the multi-view image generation device 200 provided in this embodiment are the same as those of the above embodiment. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above embodiment. In this embodiment, the multi-view image generation device 200 may include: a text determination module 220 and an image generation module 230.

[0090] The text determination module 220 is used to determine a target text, wherein the target text is used to indicate the image content of a target image.

[0091] The image generation module 230 is used to perform denoising processing on each initial image among the multiple initial images, based on the initial image and the target text, using the target generation sub-model in the trained target diffusion model to obtain the target image corresponding to the initial image, wherein the target diffusion model is trained based on multiple sample sets, each of the sample sets includes multiple sample images, overlapping area features corresponding to the multiple sample images, and sample texts, the multiple sample images in a sample set are multiple images obtained by capturing the same scene from different perspectives, the overlapping area features of each sample image are image features determined based on a local area image in a corresponding reference sample image that overlaps with the field of view of the sample image, and the reference sample image and the sample image are in the same sample set.

[0092] Please refer to Figure 8 , Figure 8The second block diagram of the multi-view image generation device 200 provided in the embodiment of the present application. In this embodiment, the multi-view image generation device 200 may further include a training module 210 .

[0093] The training module 210 is used to: obtain multiple sample image groups and sample texts corresponding to each sample image in the multiple sample image groups, wherein a sample image group includes multiple sample images of the same scene at different perspectives; for each sample image, obtain the overlapping area features corresponding to the sample image based on the sample image group to which the sample image belongs; and train the target diffusion model based on the obtained multiple sample sets, wherein each sample in the sample set includes a sample image, the overlapping area features corresponding to the sample image, and the sample text.

[0094] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 110 shown in the figure may be fixed in the operating system (OS) of the electronic device 100 and may be Figure 1 Meanwhile, the data and program codes required for executing the above modules may be stored in the memory 110.

[0095] An embodiment of the present application also provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multi-view image generation method is implemented.

[0096] In summary, the embodiments of the present application provide a multi-view image generation method, device, electronic device and readable storage medium. First, a target text for indicating the image content of a target image is obtained; then, for each of the multiple initial images, a target generation sub-model in a trained target diffusion model is used to perform denoising on the initial image based on the initial image and the target text to obtain a target image under a viewing angle corresponding to the initial image. The target diffusion model is trained based on multiple sample sets, each of which includes multiple sample images, overlapping area features corresponding to the multiple sample images, and sample texts. The multiple sample images in a sample set are multiple images obtained by capturing the same scene from different viewing angles. The overlapping area features of each sample image are image features determined based on a local area image in a corresponding reference sample image that overlaps with the field of view of the sample image. The reference sample image and the sample image are in the same sample set. In this way, by constraining the overlapping area features of multiple viewing angles in the process of training the target diffusion model, the target generation sub-model in the target diffusion model is used to generate image data consistent with multiple viewing angles, thereby reducing errors caused by scene inconsistency, and improving the performance and reliability of the autonomous driving model.

[0097] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0099] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0100] The above description is only an optional embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-view image generation method, characterized in that: The method comprises: Determining a target text, wherein the target text is used to indicate the image content of the target image; For each initial image among the multiple initial images, a target generation sub-model in a trained target diffusion model is used to perform denoising processing on the initial image based on the initial image and the target text to obtain a target image corresponding to the initial image, wherein the target diffusion model is trained based on multiple sample sets, each of the sample sets includes multiple sample images, overlapping area features corresponding to the multiple sample images, and sample texts, the multiple sample images in a sample set are multiple images obtained by capturing images of the same scene from different perspectives, the overlapping area features of each sample image are image features determined based on a local area image in a corresponding reference sample image that overlaps with the field of view of the sample image, and the reference sample image and the sample image are in the same sample set.

2. The method according to claim 1, characterized in that The method of performing denoising processing on each of the multiple initial images by using a target generation sub-model in a trained target diffusion model based on the initial image and the target text to obtain a target image corresponding to the initial image includes: Through the target generation sub-model, based on the camera parameter information corresponding to the initial image, the initial image and the target text, the target image under the viewing angle corresponding to the camera parameter information is obtained, wherein the camera parameter information corresponding to each of the multiple initial images is different, and the sample set also includes the camera parameter information corresponding to each sample image.

3. The method according to claim 1, characterized in that The target generation sub-model includes a target image encoder, a target image decoder, a target text encoder and a target denoising network. The target generation sub-model in the trained target diffusion model is used to denoise the initial image based on the initial image and the target text to obtain a target image corresponding to the initial image, including: Encoding the initial image by the target image encoder to obtain initial image features; Encoding the target text by the target text encoder to obtain text features; Using the target denoising network to denoise the initial image features according to the initial image features and text features to obtain denoised initial image features; The target image decoder is used to decode the denoised initial image features to obtain the target image.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Obtaining a plurality of sample image groups and sample texts corresponding to each sample image in the plurality of sample image groups, wherein a sample image group includes a plurality of sample images of the same scene at different viewing angles; For each sample image, according to the sample image group to which the sample image belongs, obtaining the overlapping area feature corresponding to the sample image; The target diffusion model is trained based on the obtained multiple sample sets, wherein each sample in the sample set includes a sample image, an overlapping region feature corresponding to the sample image, and a sample text.

5. The method according to claim 4, characterized in that The target diffusion model is trained based on the obtained multiple sample sets, including: For each sample in the sample set, a sample image in the sample is denoised by an initial denoising network to obtain a denoised feature map; Using an initial denoising network to denoise the noisy feature map based on sample text and overlapping area features corresponding to the sample image, and obtaining a noise loss corresponding to the sample image, wherein the initial denoising network performs denoising based on an output result of cross attention, the output result is obtained based on a Query matrix, a Key matrix, and a Value matrix, the Query matrix is ​​obtained based on the sample image, and the Key matrix and the Value matrix are obtained based on the sample text and overlapping area features corresponding to the sample image; According to the noise loss corresponding to each sample in the sample set, a total noise loss corresponding to the sample set is obtained; Adjusting parameters of an initial diffusion model according to the total noise loss, wherein the initial diffusion model includes an initial noise addition network and an initial noise removal network; When it is determined that the training stop condition is not met, the training is continued; when it is determined that the training stop condition is met, the training is stopped to obtain the target diffusion model.

6. The method according to claim 4, characterized in that The step of obtaining, for each sample image, an overlapping region feature corresponding to the sample image according to the sample image group to which the sample image belongs includes: For the sample image, determining a corresponding reference sample image from the sample image group; For each reference sample image, determining a local image corresponding to a portion of the field of view overlapping the sample image from the reference sample image; Mapping the local image to the coordinate system used by the sample image to obtain a target local image; The overlapping area features corresponding to the sample image are obtained according to the target local image.

7. The method according to claim 4, characterized in that The obtaining of the sample text corresponding to each sample image in the plurality of sample image groups includes: Encoding the sample image using a first image encoder to obtain a first sample image feature; A text generation model is used to obtain a sample text corresponding to the sample image based on the first sample image feature.

8. A multi-view image generation device, characterized in that: The device comprises: A text determination module, used to determine a target text, wherein the target text is used to indicate the image content of a target image; An image generation module is used to perform denoising processing on each initial image among multiple initial images based on the initial image and target text using a target generation sub-model in a trained target diffusion model to obtain a target image corresponding to the initial image, wherein the target diffusion model is trained based on multiple sample sets, each of the sample sets includes multiple sample images, overlapping area features corresponding to the multiple sample images, and sample text, the multiple sample images in a sample set are multiple images obtained by capturing images of the same scene from different perspectives, the overlapping area features of each sample image are image features determined based on a local area image in a corresponding reference sample image that overlaps with the field of view of the sample image, and the reference sample image and the sample image are in the same sample set.

9. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the multi-view image generation method according to any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-view image generation method as described in any one of claims 1 to 7 is implemented.

Images