Image generation method and device, electronic equipment and storage medium

Abstract

The invention relates to an image generation method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining a scene image and attribute information of a to-be-added object; determining a target environment element corresponding to the to-be-added object from a pre-designed environment element perception model; and generating a target image including the target environment element according to the attribute information and the scene image. According to the method, the target environmental element corresponding to the to-be-added object can be determined from the pre-designed environmental element perception model, so that the target image in which the target environmental element is added to the scene image can be generated according to the attribute information of the target environmental element, and the image conforming to expectation can be generated in a targeted manner, so that the user experience is improved. And the stability and controllability of image generation are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, device, electronic device and storage medium. Background Art

[0002] With the development of computer technology and camera technology, cameras are one of the most critical sensors in almost all intelligent driving perception systems. Vehicles rely on deep learning algorithms to provide vehicles with visual perception capabilities of the surrounding environment. Cameras can capture image information around the vehicle. Through image processing technology, intelligent driving systems can identify road conditions, traffic signs, pedestrians and other vehicles, thereby realizing perception of the environment. The underlying technology of current intelligent perception technology is data-driven deep learning, which assists the model in recognizing elements such as vehicles, pedestrians and obstacles by feeding the model with a large amount of training data. Although perception algorithms based on deep learning have achieved remarkable results in many fields, they usually require a large amount of labeled data to train the model. However, it is very difficult to obtain large-scale, high-quality and diverse data in some scenarios, and deep learning models are "black box" models, and it is difficult to explain their internal decision-making process from the outside. This leads to the inability to find solutions through theory when bad cases occur in the model, and can only be solved by collecting data from the corresponding scene. Therefore, how to generate controlled images in specific scenarios in a targeted manner, such as corner case data in extreme cases, has become an urgent problem to be solved. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides an image generation method, device, electronic device and storage medium.

[0004] In a first aspect, the present application provides an image generation method, the method comprising:

[0005] Get the scene image and the attribute information of the object to be added;

[0006] Determine the target environmental element corresponding to the object to be added from the pre-designed environmental element perception model;

[0007] A target image including the target environment elements is generated according to the attribute information and the scene image.

[0008] Optionally, before determining the target environmental element corresponding to the to-be-added object from a pre-designed environmental element perception model, the method further comprises: acquiring the environmental element perception model;

[0009] The process of acquiring the environmental element perception model includes:

[0010] Acquire an initial perception model; wherein the initial perception model is pre-configured with a whitelist; the whitelist includes a plurality of environmental elements, and a corresponding relationship between each of the environmental elements and a target object;

[0011] Determine the perspective transformation matrix from the installed camera to the virtual camera;

[0012] The multi-view input image captured by the installed camera is transformed according to the perspective transformation matrix to obtain a multi-view image of the virtual camera;

[0013] Performing environmental element perception on the multi-view image by using the initial perception model to obtain a first environmental element;

[0014] If the first environmental factor does not belong to the whitelist, adding the first environmental factor and the correspondence between the first environmental factor and the first target object to the whitelist;

[0015] Repeat the step of perceiving the environmental elements of the multi-view images through the initial perception model to obtain the first environmental element, until all the first environmental elements belong to the whitelist, and use the current initial perception model as the environmental element perception model.

[0016] Optionally, determining the perspective transformation matrix of the mounted camera to the virtual camera includes:

[0017] Acquire a first external parameter matrix and a first internal parameter matrix of an installed camera, and acquire a second external parameter matrix and a second internal parameter matrix of a virtual camera; the installed cameras include a plurality of cameras, and the number of the virtual cameras is the same as the number of the installed cameras;

[0018] A perspective transformation matrix from the installation camera coordinate system to the virtual camera coordinate system is determined according to the first extrinsic parameter matrix, the first intrinsic parameter matrix, the second extrinsic parameter matrix, and the second intrinsic parameter matrix.

[0019] Optionally, generating a target image including the target environment elements according to the attribute information and the scene image includes:

[0020] Converting the scene image according to the perspective transformation matrix to obtain a multi-view virtual scene image of the virtual camera;

[0021] The target image is generated using a pre-trained convolutional neural network model based on the multi-perspective virtual scene image, the target environmental elements and the attribute information.

[0022] Optionally, the training process of the convolutional neural network model includes:

[0023] Acquire a sample image set for training; wherein the sample image set includes multiple groups of target samples, the target samples include a first image and a second image; the difference image between the first image and the second image includes at least one environmental element; the second image includes the environmental element;

[0024] The following training process is performed on each group of target samples in the sample image set:

[0025] Obtain the initial parameters of the initial network model;

[0026] Performing self-attention processing on the environmental elements to obtain a weighted representation of the environmental elements;

[0027] According to the weighted representation and the initial parameters of the initial network model, performing multi-scale feature fusion of the environmental elements on the first image to obtain a third image including the environmental elements;

[0028] The loss function of the third image and the second image is calculated, and after optimizing the initial parameters, the next set of target samples is obtained from the sample image set until the loss function tends to be stable, and the initial network model is used as the final convolutional neural network model.

[0029] Optionally, transforming the scene image according to the perspective transformation matrix to obtain a multi-view virtual scene image of the virtual camera includes:

[0030] Converting the scene image according to the perspective transformation matrix to obtain a multi-view virtual image of the virtual camera;

[0031] Get the preset area;

[0032] Eliminate out-of-bounds elements in the multi-view virtual image; wherein the out-of-bounds elements do not belong to the preset area;

[0033] The coordinates of the environmental elements belonging to the preset area are normalized to obtain the multi-view virtual scene image.

[0034] Optionally, the whitelist further includes categories and positional relationships of the environmental elements; after the environmental elements are perceived by the initial perception model for the multi-view image and the first environmental element is obtained, the method further includes:

[0035] Acquire the category and position relationship of the first environmental element from the whitelist;

[0036] Obtaining a word embedding of the first environmental element;

[0037] The category, the positional relationship, and the word embedding are output to the convolutional neural network model.

[0038] In a second aspect, the present application provides an image generating device, the device comprising:

[0039] An acquisition module is used to acquire scene images and attribute information of objects to be added;

[0040] A determination module, used to determine the target environmental element corresponding to the object to be added from a pre-designed environmental element perception model;

[0041] A generation module is used to generate a target image containing the target environment elements according to the attribute information and the scene image.

[0042] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0043] Memory, used to store computer programs;

[0044] The processor is used to implement the steps of the image generation method described in any embodiment of the first aspect when executing the program stored in the memory.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the image generation method as described in any one of the embodiments of the first aspect are implemented.

[0046] Beneficial effects of this application:

[0047] The method provided by the embodiment of the present application obtains a scene image and attribute information of an object to be added; determines the target environmental element corresponding to the object to be added from a pre-designed environmental element perception model; and generates a target image containing the target environmental element based on the attribute information and the scene image. This method can determine the target environmental element corresponding to the object to be added from a pre-designed environmental element perception model, so that a target image that adds the target environmental element to the scene image can be generated based on the attribute information of the target environmental element. Since an image that meets expectations can be generated in a targeted manner, the stability and controllability of image generation are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A system architecture diagram of an image generation method provided in one embodiment of the present application;

[0051] Figure 2 A schematic diagram of a flow chart of an image generation method provided by an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a camera layout provided for an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the architecture of an image generation method provided by an embodiment of the present application;

[0054] Figure 5 A schematic diagram of a training method for an image generation model provided in one embodiment of the present application;

[0055] Figure 6 A schematic diagram of the structure of an image generating device provided by an embodiment of the present application;

[0056] Figure 7 A schematic diagram of the structure of an electronic device provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0058] The first embodiment of the present application provides an image generation method, which can be applied to Figure 1The system architecture shown in the figure includes at least a data input module 101 and a fusion module 102. The data input module 101 and the fusion module 102 establish a communication connection. The fusion module 102 generates an image according to the data input by the data input module 101. For example, the fusion module can be an image generator. Specifically, the system architecture can be a vehicle. The type of vehicle is not limited. For example, it can be a fuel vehicle, a pure electric vehicle, a hybrid vehicle, or a fuel cell vehicle, etc.

[0059] Next, based on the system architecture, the image generation method is described in detail. Figure 2 , the image generation method comprises:

[0060] Step 201: Acquire a scene image and attribute information of an object to be added.

[0061] The scene image can be an image of a certain specific scene where environmental elements need to be added, such as a scene image of a vehicle driving, and the object to be added is the environmental element that you want to add to the scene image. For example, in the process of deep learning model training, it is necessary to include pictures of certain animals in a scene where the vehicle is driving on a highway. In practice, real pictures of such extreme situations are difficult to collect, so pictures need to be generated. If a large model is used to generate pictures, although many descriptive condition restrictions are added during the generation process, the obtained pictures still have high uncertainty. In this embodiment, the scene pictures that are actually collected can be used, and the attribute information of the object to be added can be obtained. The attribute information can be the description information of the object to be added. For example, if the object to be added is a fallen bicycle, the description information can be "there is a fallen bicycle 50 meters ahead, the current vehicle is in the middle lane, and there is another lane on the left and right sides. The lane lines are white, and the fallen bicycle is on the lane line of the right lane."

[0062] Step 202: Determine the target environmental element corresponding to the object to be added from the pre-designed environmental element perception model.

[0063] In this embodiment, the target environmental elements corresponding to the object to be added can be determined from the pre-designed environmental element perception model. Taking the example of adding a fallen bicycle in a highway scene, it is necessary to search the pre-stored environmental elements of the fallen bicycle from the environmental element perception model. It should be understood that a large number of different types of environmental elements need to be added to the environmental element perception model in advance, such as lane lines, traffic signs, divergence and merging lines, curbs, water barriers, cones, vehicles, animals, etc., without limitation.

[0064] In one embodiment, before determining the target environmental element corresponding to the to-be-added object from a pre-designed environmental element perception model, the method further includes: acquiring the environmental element perception model.

[0065] Among them, the acquisition process of the environmental element perception model includes: obtaining an initial perception model; wherein the initial perception model is pre-configured with a whitelist; the whitelist includes multiple environmental elements, and the corresponding relationship between each of the environmental elements and the target object; determining the perspective transformation matrix from the installed camera to the virtual camera; converting the multi-perspective input image captured by the installed camera according to the perspective transformation matrix to obtain the multi-perspective image of the virtual camera; perceiving the environmental elements of the multi-perspective image through the initial perception model to obtain the first environmental element; if the first environmental element does not belong to the whitelist, adding the first environmental element and the corresponding relationship between the first environmental element and the first target object to the whitelist; repeating the step of perceiving the environmental elements of the multi-perspective image through the initial perception model to obtain the first environmental element, until the first environmental elements all belong to the whitelist, and using the current initial perception model as the environmental element perception model.

[0066] In this embodiment, a whitelist is configured in the pre-configured initial perception model, and the whitelist includes a variety of different types of environmental elements, such as lane lines, traffic signs, diverging and merging lines, curbs, water barriers, cones, vehicles, animals, etc. Specifically, the whitelist also stores the correspondence between each environmental element and the target object. The target object can be the name or logo of the environmental element. Of course, the whitelist can also store the category and position relationship of the environmental element, etc., without limitation. Next, determine the perspective transformation matrix from the installed camera to the virtual camera, and transform the multi-perspective input image captured by the installed camera according to the perspective transformation matrix to obtain the multi-perspective image of the virtual camera. The installed camera refers to the actual camera installed on the vehicle, and the virtual camera refers to the camera virtualized according to the layout of the installed camera. The virtual camera can convert the images taken by the cameras installed on different mass-produced vehicles to a unified perspective. The environmental elements of the multi-view images are perceived through the initial perception model, that is, the environmental elements in the multi-view images are extracted to obtain the first environmental element. If the first environmental element does not belong to the white list, the first environmental element and the correspondence between the first environmental element and the first target object are added to the white list; the step of perceiving the environmental elements of the multi-view images through the initial perception model to obtain the first environmental element is repeated until all the first environmental elements belong to the white list, and the current initial perception model is used as the environmental element perception model. Through this embodiment, all environmental elements in the collected images that are not in the environmental element perception model can be added to the environmental element perception model, thereby improving the learning ability of the environmental element perception model.

[0067] In one embodiment, determining the perspective transformation matrix from an installed camera to a virtual camera includes: obtaining a first extrinsic parameter matrix and a first intrinsic parameter matrix of the installed camera, and obtaining a second extrinsic parameter matrix and a second intrinsic parameter matrix of the virtual camera; the installed cameras include multiple cameras, and the number of virtual cameras is the same as the number of installed cameras; determining the perspective transformation matrix from the installed camera coordinate system to the virtual camera coordinate system according to the first extrinsic parameter matrix, the first intrinsic parameter matrix, the second extrinsic parameter matrix, and the second intrinsic parameter matrix.

[0068] In this embodiment, the layout diagram of installing the camera is as follows: Figure 3 , 7 cameras can be installed at different positions of the vehicle, and the first external parameter matrix and the first internal parameter matrix of each camera can be obtained. At the same time, 7 virtual cameras can be configured to obtain the second external parameter matrix and the second internal parameter matrix of the virtual camera, and the perspective transformation matrix from the installation camera coordinate system to the virtual camera coordinate system can be determined according to the first external parameter matrix, the first internal parameter matrix, the second external parameter matrix and the second internal parameter matrix. For example, the external parameter matrix T_A and the internal parameter matrix K_A of the virtual camera A, the external parameter matrix T_B and the internal parameter matrix K_B of the installation camera B, select 4 3D points in the space, use the internal and external parameter matrices to calculate the 2D coordinates mapped to the virtual camera pixel coordinate system and the installation camera pixel coordinate system respectively, and directly calculate the perspective change matrix T based on the matching relationship of the 4 sets of pixel coordinates, so that the image output by the virtual camera can be as close as possible to the image taken by the installation camera.

[0069] Step 203: Generate a target image including target environment elements according to the attribute information and the scene image.

[0070] This method can determine the target environmental element corresponding to the object to be added from a pre-designed environmental element perception model, so that a target image in which the target environmental element is added to the scene image can be generated based on the attribute information of the target environmental element. Since an image that meets expectations can be generated in a targeted manner, the stability and controllability of image generation are improved.

[0071] In one embodiment, a target image including target environmental elements is generated based on attribute information and a scene image, including: transforming the scene image according to a perspective transformation matrix to obtain a multi-perspective virtual scene image of a virtual camera; and generating the target image using a pre-trained convolutional neural network model based on the multi-perspective virtual scene image, target environmental elements and attribute information.

[0072] In this embodiment, combined with Figure 4 To illustrate with an example, Figure 4The schematic diagram of the architecture of an image generation method includes a virtual camera image conversion module, an environmental element encoder, an image encoder, an image generator, etc., wherein there are 7 installed cameras and 7 virtual cameras. The image of the installed camera is converted by the virtual camera image conversion module to obtain the corresponding virtual camera image. For example, if the image of the installed camera is a scene image, the multi-view virtual scene image of the virtual camera is converted accordingly. The multi-view virtual scene image is encoded by the image encoder and input into the image generator. The image encoder is configured with an environmental element perception model, which can perceive the environmental elements in the input image. The simulated scene perception elements to be added are input into the image generator, and the simulated scene perception elements include the attribute information of the object to be added. The image generator determines the environmental elements corresponding to the object to be added according to the object to be added. At the same time, after encoding the attribute information of the environmental elements through the environmental element encoder, a high-dimensional environmental perception feature is generated and input into the image generator. The image generator can generate the target image according to the multi-perspective virtual scene image, environmental elements and high-dimensional environmental perception features using a pre-trained convolutional neural network model. Since the expected image can be generated in a targeted manner, the stability and controllability of the image generation are improved. In addition, since the environmental element encoder is encoded based on multiple virtual cameras, it can stably generate images from multiple perspectives.

[0073] In this embodiment, the image encoder can automatically learn and extract meaningful features from the image through a neural network model to support various visual tasks, such as image classification, target detection, semantic segmentation, etc. The image encoder can accept multi-view images of a virtual camera as input, use a CNN or Transformer architecture as a feature extraction network, and generate multi-scale high-dimensional features. The environmental element encoder can obtain the categories and relative position relationships of the environmental elements in the whitelist, can accept the output of the environmental element perception module, and design an attention mechanism based on the Transformer model to extract the embedding (word embedding) of each environmental element, and output the encoded high-dimensional environmental perception features to the image generator. Taking the cone barrel as an example, the location, color and other attribute information of the cone barrel can be output. Of course, other high-dimensional environmental perception features can also be included without limitation. The image generator can use an image generation method consistent with the diffusion model diffusion to output images from the perspectives of each virtual camera. At this time, there are two approaches. One is to use the perception results of the image encoder as the input of the environmental element encoder, and directly generate images based on the coordinates of the perception results. The second is to obtain the before and after images in which specific environmental elements appear, connect the output of the image encoder to the image generator, and fuse them with the perception results of the image encoder to regenerate the image without any restrictions.

[0074] In one embodiment, the training process of the convolutional neural network model includes:

[0075] A set of sample images for training is obtained; wherein the sample image set includes multiple groups of target samples, and the target samples include a first image and a second image; the differential image of the first image and the second image includes at least one environmental element; the second image includes an environmental element; the following training process is performed on each group of target samples in the sample image set: initial parameters of the initial network model are obtained; self-attention processing is performed on the environmental elements to obtain a weighted representation of the environmental elements; multi-scale feature fusion of the environmental elements is performed on the first image according to the weighted representation and the initial parameters of the initial network model to obtain a third image including the environmental elements; the loss function of the third image and the second image is calculated, and after optimizing the initial parameters, the next set of target samples is obtained from the sample image set until the loss function tends to be stable, and the initial network model is used as the final convolutional neural network model.

[0076] In this embodiment, the sample image set for training includes multiple groups of target samples, each target sample includes a first image and a second image. For example, the first image and the second image are both acquired by a vehicle in the same scene, and the difference is that the differential image of the two differs in one or more environmental elements. For example, the first image does not include the cone barrel, and the second image includes the cone barrel, and other places in the images are the same. The schematic diagram of the training method of the image generation model is shown in FIG. Figure 5 , the virtual camera image refers to the target sample. The first image in the target sample can be input into the convolutional neural network (CNN), and the environmental elements are processed by self-attention. After obtaining the weighted representation of the environmental elements, the multi-scale feature fusion of CNN is performed by cross-attention, and the third image (including Output_Scale1, Output_Scale2, Output_Scale3) is output respectively. The loss function Loss is calculated by using Output_Scale1, Output_Scale2, and Output_Scale3 respectively with the second image in the virtual camera image. After each training of a group of samples, the parameters of the CNN model are optimized to continue the next group of training until the Loss tends to be stable, the model training is completed, and the final convolutional neural network model is obtained.

[0077] In one embodiment, a scene image is transformed according to a perspective transformation matrix to obtain a multi-perspective virtual scene image of a virtual camera, including: transforming the scene image according to a perspective transformation matrix to obtain a multi-perspective virtual image of the virtual camera; obtaining a preset area; removing out-of-bounds elements in the multi-perspective virtual image; wherein the out-of-bounds elements do not belong to the preset area; and normalizing the coordinates of environmental elements belonging to the preset area to obtain a multi-perspective virtual scene image.

[0078] In this embodiment, the perceived environmental elements can be preprocessed before being input into the environmental element encoder. The purpose of the preprocessing is to output each element into a unified coordinate system. At the same time, the environmental elements around the vehicle are unified into a preset area, out-of-bounds elements are removed, and the coordinates of all environmental elements are normalized to between [0, 1]. The environmental element encoder can use the Transformer architecture, preset N groups of query vectors query, query represents environmental elements, and the Transformer architecture also includes a key vector (Key) and a value vector (Value), where N needs to be set much larger than the number of environmental elements actually perceived, and the insufficient number is filled with 0. Q, K, and V are used to represent the input environmental elements.

[0079] In one embodiment, the whitelist also includes the categories and positional relationships of environmental elements; after the environmental elements of the multi-view image are perceived through the initial perception model and the first environmental element is obtained, the method also includes: obtaining the category and positional relationship of the first environmental element from the whitelist; obtaining the word embedding of the first environmental element; and outputting the category, positional relationship and word embedding to the convolutional neural network model.

[0080] In this embodiment, after obtaining the first environmental element by perceiving the environmental elements of the multi-view image through the initial perception model, the category and position relationship of the first environmental element can be obtained from the whitelist, and the word embedding embedding of the first environmental element can be obtained. The category, position relationship and word embedding are output to the convolutional neural network model for use in training the convolutional neural network model and generating images.

[0081] Based on the same technical concept, the second embodiment of the present application provides an image generating device, such as Figure 6 , the device comprises:

[0082] The acquisition module 601 is used to acquire the scene image and the attribute information of the object to be added;

[0083] A determination module 602 is used to determine the target environmental element corresponding to the object to be added from a pre-designed environmental element perception model;

[0084] The generating module 603 is used to generate a target image including the target environment elements according to the attribute information and the scene image.

[0085] The device can determine the target environmental element corresponding to the object to be added from a pre-designed environmental element perception model, so that a target image in which the target environmental element is added to the scene image can be generated according to the attribute information of the target environmental element. Since an image that meets expectations can be generated in a targeted manner, the stability and controllability of image generation are improved.

[0086] like Figure 7 As shown, the third embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0087] Memory 113, used for storing computer programs;

[0088] In one embodiment, the processor 111 is configured to implement the image generation method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113 .

[0089] The memory and processor in the above electronic device communicate through the communication bus and the communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0090] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0091] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0092] A fourth embodiment of the present application provides a computer-readable medium having a non-volatile program code executable by a processor.

[0093] Optionally, in an embodiment of the present application, a computer-readable medium is configured to store program code for a processor to execute the above method.

[0094] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0095] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0096] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present application, or a combination thereof.

[0097] For software implementation, the technology of this article can be implemented by a unit that performs the functions of this article. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0098] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0100] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0103] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0104] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0105] The above embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present application is within the protection scope of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Get the scene image and the attribute information of the object to be added; Determining the target environmental element corresponding to the object to be added from a pre-designed environmental element perception model; A target image including the target environment elements is generated according to the attribute information and the scene image.

2. The method according to claim 1, characterized in that Before determining the target environmental element corresponding to the to-be-added object from a pre-designed environmental element perception model, the method further comprises: acquiring the environmental element perception model; The process of acquiring the environmental element perception model includes: Acquire an initial perception model; wherein the initial perception model is pre-configured with a whitelist; the whitelist includes a plurality of environmental elements, and a corresponding relationship between each of the environmental elements and a target object; Determine the perspective transformation matrix from the installed camera to the virtual camera; The multi-view input image captured by the installed camera is transformed according to the perspective transformation matrix to obtain a multi-view image of the virtual camera; Performing environmental element perception on the multi-view image by using the initial perception model to obtain a first environmental element; If the first environmental factor does not belong to the whitelist, adding the first environmental factor and the correspondence between the first environmental factor and the first target object to the whitelist; Repeat the step of perceiving the environmental elements of the multi-view images through the initial perception model to obtain the first environmental element, until all the first environmental elements belong to the whitelist, and use the current initial perception model as the environmental element perception model.

3. The method according to claim 2, characterized in that Determine the perspective transformation matrix from the installed camera to the virtual camera, including: Acquire a first external parameter matrix and a first internal parameter matrix of an installed camera, and acquire a second external parameter matrix and a second internal parameter matrix of a virtual camera; the installed cameras include a plurality of cameras, and the number of the virtual cameras is the same as the number of the installed cameras; A perspective transformation matrix from the installation camera coordinate system to the virtual camera coordinate system is determined according to the first extrinsic parameter matrix, the first intrinsic parameter matrix, the second extrinsic parameter matrix, and the second intrinsic parameter matrix.

4. The method according to claim 2, characterized in that: Generating a target image including the target environment elements according to the attribute information and the scene image, comprising: Converting the scene image according to the perspective transformation matrix to obtain a multi-view virtual scene image of the virtual camera; The target image is generated using a pre-trained convolutional neural network model based on the multi-perspective virtual scene image, the target environmental elements and the attribute information.

5. The method according to claim 4, characterized in that The training process of the convolutional neural network model includes: Acquire a sample image set for training; wherein the sample image set includes multiple groups of target samples, the target samples include a first image and a second image; the difference image between the first image and the second image includes at least one environmental element; the second image includes the environmental element; The following training process is performed on each group of target samples in the sample image set: Obtain the initial parameters of the initial network model; Performing self-attention processing on the environmental elements to obtain a weighted representation of the environmental elements; According to the weighted representation and the initial parameters of the initial network model, performing multi-scale feature fusion of the environmental elements on the first image to obtain a third image including the environmental elements; The loss function of the third image and the second image is calculated, and after optimizing the initial parameters, the next set of target samples is obtained from the sample image set until the loss function tends to be stable, and the initial network model is used as the final convolutional neural network model.

6. The method according to claim 4, characterized in that The scene image is transformed according to the perspective transformation matrix to obtain a multi-view virtual scene image of the virtual camera, including: Converting the scene image according to the perspective transformation matrix to obtain a multi-view virtual image of the virtual camera; Get the preset area; Eliminate out-of-bounds elements in the multi-view virtual image; wherein the out-of-bounds elements do not belong to the preset area; The coordinates of the environmental elements belonging to the preset area are normalized to obtain the multi-view virtual scene image.

7. The method according to claim 4, characterized in that The whitelist also includes the categories and location relationships of the environmental elements; After obtaining the first environmental element by performing environmental element perception on the multi-view image through the initial perception model, the method further includes: Acquire the category and position relationship of the first environmental element from the whitelist; Obtaining a word embedding of the first environmental element; The category, the positional relationship, and the word embedding are output to the convolutional neural network model.

8. An image generating device, characterized in that: The device comprises: An acquisition module is used to acquire scene images and attribute information of objects to be added; A determination module, used to determine the target environmental element corresponding to the object to be added from a pre-designed environmental element perception model; A generation module is used to generate a target image containing the target environment elements according to the attribute information and the scene image.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images