Image generation method and device, electronic equipment and storage medium

Abstract

The invention discloses an image generation method and device, electronic equipment and a storage medium, and belongs to the technical field of electronic equipment. The image generation method comprises the steps that cue word information and a mask image corresponding to a first image are acquired, the cue word information is description information of an image generation task corresponding to the first image, and the mask image is a mask image of a reference image under the shooting view angle of the first image; and inputting the first image, the cue word information and the mask image into a diffusion model to obtain a second image.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to an image generation method, device, electronic equipment and storage medium. Background Art

[0002] Using electronic devices to take natural scenery photos is one of the common application scenarios in electronic device photography. However, natural scenery photography is easily affected by factors such as weather, light, visibility, and light brightness. For example, when taking pictures of the starry sky at night, the imaging effect of electronic devices is poor due to the low visibility at night, so this type of photography is very difficult.

[0003] In the related art, when an electronic device performs natural scenery photography, after capturing an image, the image can be adjusted based on pixel information in the image, but the adjustment based only on pixel information cannot meet the user's shooting needs. Therefore, the image quality of the image captured by the electronic device is poor. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image generation method, device, electronic device and storage medium, which can improve the image quality of images taken by the electronic device.

[0005] In a first aspect, an embodiment of the present application provides an image generation method, the method comprising:

[0006] Obtain prompt word information and a mask image corresponding to the first image, where the prompt word information is description information of the image generation task corresponding to the first image, and the mask image is a mask image of the reference image under the shooting angle of the first image; input the first image, the prompt word information and the mask image into the diffusion model to obtain a second image.

[0007] In a second aspect, an embodiment of the present application provides an image generation device, which includes: an acquisition module and a processing module. The acquisition module is used to acquire prompt word information and a mask image corresponding to a first image, wherein the prompt word information is the description information of the image generation task corresponding to the first image, and the mask image is the mask image of the reference image under the shooting angle of the first image. The processing module is used to input the first image, the prompt word information and the mask image into a diffusion model to obtain a second image.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the image generation method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the image generation method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the image generation method as described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the image generation method as described in the first aspect.

[0012] In the embodiment of the present application, the prompt word information and the mask image corresponding to the first image are obtained, the prompt word information is the description information of the image generation task corresponding to the first image, and the mask image is the mask image of the reference image under the shooting angle of the first image. The first image, the prompt word information and the mask image are input into the diffusion model to obtain the second image. In this way, since the diffusion model itself has the function of image optimization, and the above method uses the prompt word information, and the prompt word information is related to the image generation task, and the image generation task is generated according to user needs, the obtained second image is more in line with user needs. The above method also uses the mask image in the image optimization process, so that the image optimization process pays more attention to the key area corresponding to the mask image in the first image, and since the mask image is the mask image of the reference image, and the reference image corresponds to the real physical world, the optimization of the first image is based on the optimization of the real physical world, which is more in line with the actual situation. Therefore, the second image obtained by this scheme has better quality, is more in line with the real physical world, meets user needs, and improves the image quality of the image taken by the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is one of the flowcharts of the image generation method provided in the embodiment of the present application;

[0014] Figure 2 This is the second flow chart of the image generation method provided in the embodiment of the present application;

[0015] Figure 3 This is the third flow chart of the image generation method provided in the embodiment of the present application;

[0016] Figure 4 This is the fourth flowchart of the image generation method provided in the embodiment of the present application;

[0017] Figure 5This is the fifth flowchart of the image generation method provided in the embodiment of the present application;

[0018] Figure 6 This is the sixth flowchart of the image generation method provided in the embodiment of the present application;

[0019] Figure 7 This is the seventh flow chart of the image generation method provided in the embodiment of the present application;

[0020] Figure 8 This is the eighth flowchart of the image generation method provided in the embodiment of the present application;

[0021] Fig. 9 This is the ninth flowchart of the image generation method provided in the embodiment of the present application;

[0022] Fig.10 is a schematic diagram of an image generating device provided in an embodiment of the present application;

[0023] Fig.11 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0024] Fig.12 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0027] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0028] The image generation method, device, electronic device and storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0029] The image generation method provided in the embodiment of the present application can be applied to a scene where an electronic device shoots natural scenery. Exemplarily, the scene where an electronic device shoots natural scenery may include a scene where an electronic device shoots a starry sky image, a scene where an electronic device shoots a mountain, a river, a scene where an electronic device shoots a natural scenery of the changing seasons, etc. Taking the scene where an electronic device shoots a starry sky image as an example, the starry sky image may include stars or the moon.

[0030] The embodiment of the present application provides an image generation method, obtaining prompt word information and a mask image corresponding to a first image, the prompt word information is the description information of the image generation task corresponding to the first image, and the mask image is the mask image of the reference image under the shooting angle of the first image. The first image, the prompt word information and the mask image are input into the diffusion model to obtain the second image. In this way, since the diffusion model itself has the function of image optimization, and the above method uses the prompt word information, and the prompt word information is related to the image generation task, and the image generation task is generated according to user needs, the obtained second image is more in line with user needs. The above method also uses the mask image in the image optimization process, so that the image optimization process pays more attention to the key area corresponding to the mask image in the first image, and since the mask image is the mask image of the reference image, and the reference image corresponds to the real physical world, the optimization of the first image is based on the optimization of the real physical world, which is more in line with the actual situation. Therefore, the second image obtained by this scheme has better quality, is more in line with the real physical world, meets user needs, and improves the image quality of the image taken by the electronic device.

[0031] The execution subject of the image generation method provided in the embodiment of the present application may be an image generation device. Exemplarily, the image generation device may be an electronic device, or a functional component or functional entity in the electronic device. The image generation method provided in the embodiment of the present application will be described exemplarily below by taking the execution subject as an electronic device as an example.

[0032] Figure 1 is a flow chart of the image generation method provided in the embodiment of the present application, such as Figure 1 As shown, the image generation method provided in the embodiment of the present application may include the following steps 101 and 102.

[0033] Step 101: The electronic device obtains prompt word information and a mask image corresponding to a first image.

[0034] In the embodiment of the present application, the first image may be a natural scenery image. For example, the first image may be any natural scenery image such as a starry sky image, a mountain and river image, a sea of ​​clouds image, a desert image, a sunrise image, a sunset image, or images of these natural scenery at different seasons.

[0035] Optionally, in an embodiment of the present application, the electronic device obtaining the first image may include: the electronic device may capture the first image in real time, or the electronic device may determine the first image to be processed from pre-stored images based on user selection, or the electronic device may receive an image input by the user and determine the image input by the user as the first image.

[0036] In the embodiment of the present application, the prompt word information is the description information of the image generation task corresponding to the first image.

[0037] Optionally, in an embodiment of the present application, the image generation task corresponding to the first image may be generated according to user needs, and therefore, the prompt word information may be understood as description information for describing the image obtained according to user needs.

[0038] Exemplarily, the prompt word information may be "generate a picture of a starry night sky" or "generate a picture of a starry sky containing many stars in the Milky Way" or "generate an image of a desert with yellow sand all over the sky" or "generate an image of rolling mountains and rivers" and so on.

[0039] Optionally, in an embodiment of the present application, image processing software may be installed in the electronic device, and the user may input prompt word information through an input control on an interface of the image processing software.

[0040] In the embodiment of the present application, the mask image is a mask image of a reference image under the shooting angle of the first image.

[0041] Optionally, in an embodiment of the present application, the above-mentioned shooting angle may include position information and posture information of the electronic device.

[0042] Optionally, in an embodiment of the present application, when the first image is an image taken in real time by the electronic device, the electronic device can utilize the positioning function of the electronic device to obtain its own position information when taking the first image, and obtain the posture of the electronic device when taking the first image based on the gyroscope data of the gyroscope configured for the electronic device, and calculate the direction and pitch angle of the camera of the electronic device.

[0043] Optionally, in an embodiment of the present application, the electronic device may be configured with a hardware or software system capable of implementing a positioning function, and positioning can be achieved through the above hardware or software.

[0044] Exemplarily, the software system capable of realizing the positioning function configured in the above-mentioned electronic device may be a global positioning system (GPS) or a Beidou positioning system, and the like.

[0045] Optionally, in an embodiment of the present application, the orientation of the camera of the above-mentioned electronic device may include an angle of north-east or an angle of north-west, and the angle may be 0 degrees to 180 degrees, where 0 degrees represents due north and 180 degrees represents due south.

[0046] Alternatively, the orientation of the camera of the electronic device may include an angle of south-east or south-west, and the angle may be 0 degrees to 180 degrees, where 0 degrees represents due south and 180 degrees represents due north.

[0047] Alternatively, the orientation of the camera of the electronic device may include an angle of east by north or east by south, and the angle may be 0 degrees to 180 degrees, where 0 degrees represents due east and 180 degrees represents due west.

[0048] Alternatively, the orientation of the camera of the electronic device may include an angle of west by north or west by south, and the angle may be 0 degrees to 180 degrees, where 0 degrees represents due west and 180 degrees represents due east.

[0049] For example, if the direction of the camera of the electronic device is 30 degrees north by east, the camera of the electronic device is 30 degrees north by east. Alternatively, if the direction of the camera of the electronic device is 0 degrees south by east, the camera of the electronic device is oriented towards the south.

[0050] Optionally, in an embodiment of the present application, the pitch angle of the camera of the above-mentioned electronic device includes 0 degrees to 90 degrees, where 0 degrees represents horizontal front and 90 degrees represents vertical upward.

[0051] For example, if the pitch angle of the camera of the electronic device is 90 degrees, the pitch angle of the camera of the electronic device is vertically upward. Alternatively, if the pitch angle of the camera of the electronic device is 45 degrees, the pitch angle of the camera of the electronic device is an elevation angle of 45 degrees.

[0052] Optionally, in the embodiment of the present application, when the first image is an image taken by the electronic device in the past, the electronic device may store the position information and posture information when the first image was taken. Alternatively, when the first image is an image input by the user, the user may also input the position information and posture information of the electronic device that captured the first image when the first image was captured.

[0053] Optionally, in an embodiment of the present application, the reference image may be understood as a natural scenery image having the same shooting angle as the first image and higher image quality.

[0054] Optionally, in an embodiment of the present application, in the case where the above-mentioned first image is a captured starry sky image, the above-mentioned reference image may be a star map or a celestial map. Among them, a star map may also be referred to as a map of the night sky, that is, a "map of stars". Typically, astronomers use grids to divide the stars to make star maps easier to use, and star maps are typically used to identify and locate stars, constellations, galaxies, and the like. Alternatively, in the case where the above-mentioned first image is a captured mountain and river image, the above-mentioned reference image may be a mountain and river map with the same shooting angle as the first image and higher image quality.

[0055] Optionally, in the embodiment of the present application, the electronic device acquiring the mask image may include: the electronic device acquiring a reference image under the shooting angle of the first image from a reference gallery, and then performing mask processing on the reference image to obtain the mask image.

[0056] Taking the first image as a starry sky image as an example, the electronic device can obtain a star map under the shooting angle of the starry sky image from the star map library, determine the positions of all stars in the star map, and generate a mask image according to different brightness levels of the stars. It should be noted that the higher the brightness of the star, the larger the area corresponding to the star in the mask image, and the lower the brightness of the star, the smaller the area corresponding to the star in the mask image.

[0057] For example, the electronic device may set the values ​​of all pixels in the area corresponding to each star in the mask image to 1, and the values ​​of other pixels to 0. Alternatively, the electronic device may set the values ​​of all pixels in the area corresponding to stars of different brightness levels from 1 to 0 according to the brightness level from high to low.

[0058] For example, the electronic device sets the values ​​of all pixels in the area corresponding to the star with the highest brightness level to 1, sets the values ​​of all pixels in the area corresponding to the star with a higher brightness level to 0.6, sets the values ​​of all pixels in the area corresponding to the star with a lower brightness level to 0.4, and sets the values ​​of all pixels in the area corresponding to the star with the lowest brightness level to 0.

[0059] Taking the first image as a mountain image as an example, the electronic device can obtain a mountain image from the mountain image library under the shooting angle of the mountain image, and then mask the main area or foreground area in the mountain image to generate a mask image. The main area can be a mountain or a tree or other content. Exemplarily, the electronic device can set the value of all pixels in the main area of ​​the mountain image to 1 and the value of other pixels to 0 to obtain a mask image.

[0060] Step 102: The electronic device inputs the first image, the prompt word information and the mask image into a diffusion model to obtain a second image.

[0061] Optionally, in the embodiment of the present application, the above-mentioned diffusion model can be any model with image generation function and diffusion generation function.

[0062] For example, the diffusion model may be a stable diffusion (SD) model, such as an SD1.5 model or other derivative models of SD1.5.

[0063] Optionally, in the embodiment of the present application, the above prompt word information can be represented in the form of a feature vector.

[0064] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 2 As shown, the above step 102 can be implemented by following steps 201 and 202.

[0065] Step 201: The electronic device performs feature processing on the first image, the prompt word information and the mask image through N cross attention modules in the diffusion model to obtain a first implicit feature map.

[0066] In the embodiment of the present application, the first implicit feature map represents the position of the feature pixel points in the first image in the implicit space. N is an integer greater than 1.

[0067] Optionally, in the embodiment of the present application, the above-mentioned characteristic pixel points can be understood as pixel points included in the area that needs to be focused on in the first image.

[0068] Taking the first image as a starry sky image as an example, the above-mentioned characteristic pixel points may be pixel points corresponding to stars or the moon in the starry sky image, and the positions of the above-mentioned characteristic pixel points are the positions of stars or the moon in the starry sky image. Therefore, the above-mentioned first implicit feature map represents the positions of stars or the moon in the starry sky image in the implicit space.

[0069] Taking the first image as a mountain image as an example, the above-mentioned characteristic pixel points can be pixel points included in the main area or the foreground area in the mountain image, and the position of the above-mentioned characteristic pixel points is the position of the main area or the foreground area in the mountain image. Therefore, the above-mentioned first implicit feature map represents the position of the main area or the foreground area in the mountain image in the implicit space.

[0070] It can be understood that the above-mentioned first implicit feature map is an image obtained by combining the mask image and the prompt word information.

[0071] Optionally, in an embodiment of the present application, the above-mentioned cross-attention module may be a cross-attention module.

[0072] Optionally, in the embodiment of the present application, combined with Figure 2 ,like Figure 3 As shown, the above step 201 can be implemented through the following steps 2011 to 2013.

[0073] Step 2011: The electronic device performs linear transformation processing on the target image through the i-th cross-attention module among N cross-attention modules to obtain the i-th query vector Q, and performs linear transformation processing on the prompt word information to obtain the i-th key vector K and the i-th value vector V.

[0074] Optionally, in an embodiment of the present application, the query vector Q may be an image Query vector, the key vector K may be a prompt word Key vector, and the value vector V may be a prompt word Value vector.

[0075] Optionally, in the embodiment of the present application, each cross attention module may include three feature matrices, namely, the first feature matrix W Q (i) , the second characteristic matrix W K (i) and the third characteristic matrix W V (i) Furthermore, the first feature matrices in different cross-attention modules may be different, the second feature matrices in different cross-attention modules may be different, and the third feature matrices in different cross-attention modules may be different.

[0076] Where i represents the i-th cross attention module, Q represents the Query vector, K represents the Key vector, and V represents the Value vector. Q (i) represents the first feature matrix in the i-th criss-cross attention module, W K (i) represents the second feature matrix in the i-th criss-cross attention module, W V (i)represents the third feature matrix in the i-th criss-cross attention module.

[0077] Optionally, in an embodiment of the present application, the electronic device may calculate the query vector Q by the following formula (1), calculate the key vector K by the following formula (2), and calculate the value vector V by the following formula (3).

[0078] Q i =W Q (i) ·z i (1)

[0079] K i =W K (i) ·τ (2)

[0080] V i =W V (i) ·τ (3)

[0081] Among them, i represents the i-th cross attention module, z i represents the target image, τ represents the prompt word information, Q i represents the i-th query vector Q, K i represents the i-th key vector K, V i Represents the i-th value vector V.

[0082] In the embodiment of the present application, when i=1, the target image is the first image. When i=2, ..., N, the target image is the i-1th intermediate feature map.

[0083] Optionally, in an embodiment of the present application, the above-mentioned i-1th intermediate feature map is the output of the i-1th cross-attention module among the above-mentioned N cross-attention modules.

[0084] Step 2012: The electronic device calculates the i-th intermediate feature map through the i-th cross-attention module based on the i-th query vector Q, the i-th key vector K, the i-th value vector V and the mask image.

[0085] Optionally, in an embodiment of the present application, the above step 2012 can be implemented by following the steps 20121 to 20123.

[0086] Step 20121: The electronic device calculates the product of the ith query vector Q and the ith key vector K through the ith cross-attention module to obtain a first result.

[0087] Step 20122: The electronic device normalizes the first result through the i-th cross-attention module to obtain a second result.

[0088] Step 20123: The electronic device calculates the product of the second result, the mask image and the i-th value vector V through the i-th cross-attention module to obtain the i-th intermediate feature map.

[0089] Exemplarily, the electronic device may calculate the i-th intermediate feature map by the following formula (4).

[0090]

[0091] Among them, B represents the mask image, d represents K i The dimension, Q i K i Indicates the first result, represents the second result, and A represents the i-th intermediate feature map.

[0092] In this way, the mask image is added to the calculation of the cross-attention module to focus on the positions where the stars appear. At the same time, stars of different brightness levels are represented by different pixel values ​​in the mask image, and the obtained intermediate feature map also pays different attention to stars of different brightness levels.

[0093] Step 2013: After obtaining the Nth intermediate feature map, the electronic device iterates the feature processing M times, and uses the last intermediate feature map obtained after iterating M times as the first implicit feature map.

[0094] It should be noted that the above M times may be a preset number of times, for example, the above M times may be 30 times.

[0095] Optionally, in an embodiment of the present application, the last intermediate feature map obtained after the above iteration M times is the intermediate feature map output by the last cross-attention module after iteration M times, that is, the Nth cross-attention module.

[0096] Exemplarily, taking the diffusion model including 3 cross-attention modules, that is, the above N is 3, and M is 30 times, that is, the above M is 30 as an example, the process of obtaining the first implicit feature map is described in detail. The electronic device inputs the first image, the prompt word information and the mask image into the first cross-attention module of the diffusion model. In the first cross-attention module, the electronic device uses the above formula (1) to perform linear transformation processing on the first image to obtain the first query vector Q, and uses the above formula (2) and formula (2) to perform linear transformation processing on the prompt word information to obtain the first key vector K and the first value vector V respectively. Then, based on the first query vector Q, the first key vector K, the first value vector V and the mask image, the above formula (4) is used to calculate the first intermediate feature map. The electronic device then inputs the first intermediate feature map, the prompt word information and the mask image into the second cross-attention module. In the second cross-attention module, the electronic device uses the above formula (1) to perform linear transformation processing on the first intermediate feature map to obtain the second query vector Q, and uses the above formula (2) and formula (2) to perform linear transformation processing on the prompt word information to obtain the second key vector K and the second value vector V respectively. Then, based on the second query vector Q, the second key vector K, the second value vector V and the mask image, the above formula (4) is used to calculate the second intermediate feature map. The electronic device then inputs the second intermediate feature map, the prompt word information and the mask image into the third cross-attention module. In the third cross-attention module, the electronic device uses the above formula (1) to perform linear transformation processing on the second intermediate feature map to obtain the third query vector Q, and uses the above formula (2) and formula (2) to perform linear transformation processing on the prompt word information to obtain the third key vector K and the third value vector V respectively. Then, based on the third query vector Q, the third key vector K, the third value vector V and the mask image, the above formula (4) is used to calculate the third intermediate feature map. Then, the third intermediate feature map is input into the first cross-attention module, and the above operation is iterated 30 times until the third intermediate feature map is obtained for the 30th time, and the third intermediate feature map obtained for the 30th time is used as the first implicit feature map.

[0097] Step 202: The electronic device decodes the first implicit feature map to obtain a second image.

[0098] Optionally, in an embodiment of the present application, the electronic device may input the first implicit feature map into an image decoder, map the first implicit feature map from the implicit space to the pixel space, and obtain a second image.

[0099] Exemplarily, the image decoder may be any decoder capable of mapping an image from an implicit space to a pixel space. For example, the image decoder may be a VAE-Decoder.

[0100] It can be understood that the first implicit feature image is a representation of the second image in the implicit space, and the second image is a representation of the first implicit feature image in the pixel space.

[0101] For example, see Figure 4 The electronic device inputs the first image, the prompt word information and the mask image into the image generation model, and after processing through the diffusion model of the image generation model (see steps 2011 to 2013 above), a first implicit feature map can be obtained, and then the first implicit feature map is input into the image decoder of the image generation model for decoding to obtain a second image.

[0102] In this way, under the guidance of the prompt word information and the mask image, the electronic device generates stars in specific areas of the first image or processes stars in specific areas of the first image through the modified attention calculation process in the cross-attention module, and finally achieves the generation of high-quality physically consistent starry sky images, thereby improving the image quality of images taken by the electronic device.

[0103] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 5 As shown, the above step 102 can be implemented by following the steps 301 to 305.

[0104] Step 301: The electronic device performs feature processing on the first image and prompt word information through L cross attention modules in the diffusion model to obtain L first feature maps.

[0105] In this embodiment of the present application, each cross-attention module corresponds to a first feature map, and L is an integer greater than 1.

[0106] Optionally, in an embodiment of the present application, the electronic device performs linear transformation processing on the target image through the i-th cross-attention module among L cross-attention modules to obtain the i-th query vector Q, and performs linear transformation processing on the prompt word information to obtain the i-th key vector K and the i-th value vector V, and then based on the i-th query vector Q, the i-th key vector K and the i-th value vector V, calculates the i-th first feature map.

[0107] Optionally, in the embodiment of the present application, when i=1, the target image is the first image. When i=2, ..., N, the target image is the i-1th first feature image.

[0108] Exemplarily, the electronic device can calculate the i-th query vector Q, the i-th key vector K and the i-th value vector V by the above formulas (1), (2) and (3), respectively, and then calculate the i-th first feature map according to the i-th query vector Q, the i-th key vector K and the i-th value vector V by the following formula (5).

[0109]

[0110] Where d represents K i The dimension of , C represents the i-th first feature map.

[0111] Step 302: The electronic device calculates a first difference value based on the L first feature maps and the mask image by using a difference calculation function.

[0112] In the embodiment of the present application, the first difference value represents the difference between the L first feature maps and the mask image, and the smaller the difference, the better the model effect.

[0113] Optionally, in the embodiment of the present application, the difference calculation function may be L1=L FG +L BG .

[0114] Optionally, in the embodiment of the present application, the above L FG =Σ i∈N (1-max(A i B)), the above L BG =∑ i∈N max(A i ·(1-B)).

[0115] Among them, A i represents the first feature map of the i-th order, max(A i B) represents the maximum pixel product among multiple pixel products obtained by multiplying the i-th first feature map and the mask image pixel by pixel. max(A i (1-B)) represents the largest pixel product among multiple pixel products obtained by multiplying the ith first feature map by the difference between 1 and the mask image.

[0116] Optionally, in an embodiment of the present application, in the mask image, the value of the pixel points in the area with stars is 1, and the value of the pixel points in the area without stars is 0. Therefore, after the i-th first feature map is multiplied pixel by pixel by pixel, the largest pixel product is still equal to the value of the pixel point in the first feature map.

[0117] Optionally, in an embodiment of the present application, the above-mentioned L first feature maps and the mask image are substituted into the above-mentioned difference calculation function to calculate the first difference value.

[0118] Step 303: When the first difference value is greater than the first threshold, the electronic device iteratively updates the characteristic graph parameters in the difference calculation function until the difference value obtained by the difference calculation function is less than or equal to the first threshold.

[0119] It should be noted that the above-mentioned first threshold can be set by default, or can be set and adjusted according to actual needs, and the embodiments of the present application are not limited to this.

[0120] Optionally, in an embodiment of the present application, the electronic device may iteratively update the characteristic graph parameters in the difference calculation function by using the following formula (6).

[0121]

[0122] in, represents the feature map parameters after iterative update, x t represents the Nth first feature map parameter after iterative update, α represents the adjustable step size, Represents the gradient descent value of the difference calculation function.

[0123] Exemplarily, the electronic device may calculate N feature map parameters by gradient descent, and the N feature map parameters are such that the difference value calculated by the difference calculation function is less than or equal to the first threshold value, and the last one of the N feature map parameters is determined as x t , and then the iteratively updated feature map parameters are calculated using the above formula (6).

[0124] Step 304: The electronic device iterates feature processing Y times through L cross attention modules in the diffusion model based on the iteratively updated feature map parameters and prompt word information, and uses the last feature map obtained after Y iterations as the second implicit feature map.

[0125] In the embodiment of the present application, the second implicit feature map represents the positions of feature pixel points in the first image in the implicit space.

[0126] It should be noted that Y is an integer greater than 1, and Y times can be a preset number of times, and the above Y times can be the same as or different from the above M times, which is not limited in the embodiment of the present application. For example, the above Y times can be 26 times.

[0127] Optionally, in an embodiment of the present application, the electronic device returns to iteratively execute the above steps 301 to 303 through the above L cross-attention modules, replaces the first image with the iteratively updated feature map parameters, and after iterative processing Y times, uses the feature map output by the last cross-attention module as the second implicit feature map.

[0128] Step 305: The electronic device decodes the second implicit feature map to obtain a second image.

[0129] It should be noted that the implementation process of step 305 can refer to the relevant description of step 202 above, and this embodiment will not be repeated here.

[0130] For example, see Figure 6 , Figure 6 "A" in the figure is the first characteristic diagram in the above embodiment, Figure 6 "B" in the above embodiment is the mask image, Figure 6 Xt in the above embodiment is the second implicit feature map. The electronic device inputs the first image, the prompt word information and the mask image into the image generation model, and after processing through the diffusion model of the image generation model (see the above steps 301 to 304), the second implicit feature map can be obtained, and then the second implicit feature map is input into the image decoder of the image generation model for decoding to obtain the second image.

[0131] Optionally, in the embodiment of the present application, after the above step 302, the image generation method provided in the embodiment of the present application may further include the following step 306.

[0132] Step 306: When the first difference value is less than or equal to the first threshold, the electronic device obtains a second image based on the last feature map in the L first feature maps, the prompt word information and the mask image through L cross attention modules in the diffusion model.

[0133] In an embodiment of the present application, the last feature map mentioned above is a feature map output by the last cross-attention module among the L cross-attention modules mentioned above.

[0134] Optionally, in an embodiment of the present application, when the first difference value is less than or equal to the first threshold value, the electronic device performs feature processing on the last feature map and the prompt word information in the L first feature maps through the L cross-attention modules in the diffusion model to obtain L second feature maps. The electronic device calculates the second difference value based on the L second feature maps and the mask image through the difference calculation function. When the second difference value is greater than the first threshold value, the feature map parameters in the difference calculation function are iteratively updated until the difference value obtained by the difference calculation function is less than or equal to the first threshold value. Through the L cross-attention modules in the diffusion model, based on the iteratively updated feature map parameters and the prompt word information, the feature processing is iterated Y times, and the last feature map obtained after the Y iterations is used as the second implicit feature map, and the second implicit feature map is decoded to obtain a second image.

[0135] Exemplarily, taking the diffusion model including two cross-attention modules, i.e., L is 2, and Y is 20, i.e., Y is 20, as an example, the process of obtaining the second image is described in detail. The electronic device inputs the first image and the prompt word information into the first cross-attention module of the diffusion model. In the first cross-attention module, the electronic device uses the above formula (1) to perform linear transformation processing on the first image to obtain the first query vector Q, and uses the above formula (2) and formula (2) to perform linear transformation processing on the prompt word information to obtain the first key vector K and the first value vector V respectively. Then, based on the first query vector Q, the first key vector K and the first value vector V, the above formula (5) is used to calculate the first first feature map. The electronic device then inputs the first first feature map and the prompt word information into the second cross-attention module. In the second cross-attention module, the electronic device uses the above formula (1) to perform linear transformation processing on the first first feature map to obtain the second query vector Q, and uses the above formula (2) and formula (2) to perform linear transformation processing on the prompt word information to obtain the second key vector K and the second value vector V respectively, and then calculates the second first feature map based on the second query vector Q, the second key vector K and the second value vector V using the above formula (5). The electronic device calculates the first difference value based on the two first feature maps and the mask image through the difference calculation function. When the first difference value is less than or equal to the first threshold, the electronic device inputs the second first feature map and the prompt word information into the first cross-attention module of the diffusion model. In the first cross-attention module, the electronic device uses the above formula (1) to perform linear transformation processing on the second first feature map to obtain the first query vector Q, and uses the above formula (2) and formula (2) to perform linear transformation processing on the prompt word information to obtain the first key vector K and the first value vector V respectively, and then based on the first query vector Q, the first key vector K and the first value vector V, the above formula (5) is used to calculate the first second feature map. The electronic device then inputs the first second feature map and the prompt word information into the second cross-attention module. In the second cross-attention module, the electronic device uses the above formula (1) to perform linear transformation processing on the first second feature map to obtain the second query vector Q, and uses the above formula (2) and formula (2) to perform linear transformation processing on the prompt word information to obtain the second key vector K and the second value vector V respectively, and then based on the second query vector Q, the second key vector K and the second value vector V, the above formula (5) is used to calculate the second second feature map. The electronic device calculates the second difference value based on the two second feature maps and the mask image through the difference calculation function. When the second difference value is greater than the first threshold, the electronic device iteratively updates the feature map parameters in the difference calculation function until the difference value obtained by the difference calculation function is less than or equal to the first threshold.The electronic device inputs the iteratively updated feature map parameters and prompt word information into the first cross-attention module of the diffusion model. In the first cross-attention module, the electronic device uses the above formula (1) to perform linear transformation processing on the iteratively updated feature map parameters to obtain the first query vector Q, and uses the above formula (2) and formula (2) to perform linear transformation processing on the prompt word information to obtain the first key vector K and the first value vector V respectively. Then, based on the first query vector Q, the first key vector K and the first value vector V, the above formula (5) is used to calculate the first third feature map. The electronic device then inputs the first third feature map and the prompt word information into the second cross-attention module. In the second cross-attention module, the electronic device uses the above formula (1) to perform linear transformation processing on the first third feature map to obtain the second query vector Q, and uses the above formula (2) and formula (2) to perform linear transformation processing on the prompt word information to obtain the second key vector K and the second value vector V respectively. Then, based on the second query vector Q, the second key vector K and the second value vector V, the above formula (5) is used to calculate the second third feature map. The electronic device calculates the third difference value based on the two third feature maps and the mask image through the difference calculation function. By analogy, the above operation is iterated 20 times in total until the second twentieth feature map is obtained, and the second twentieth feature map is used as the second implicit feature map. The electronic device then decodes the second implicit feature map to obtain the second image.

[0136] Optionally, in the embodiment of the present application, before step 102, the image generation method provided in the embodiment of the present application may further include the following step 103, and the above step 102 may be implemented by the following step 1021.

[0137] Step 103: The electronic device encodes and adds noise to the first image to obtain a third image.

[0138] Optionally, in an embodiment of the present application, the electronic device may first encode the first image, and then perform noise processing on the encoding result to obtain a third image.

[0139] Optionally, in an embodiment of the present application, the electronic device may input the first image into an image encoder for encoding, and in the image encoder, the first image is mapped from a pixel space to an implicit space, and the encoding result obtained is an implicit representation (latent image) of the first image. Noise of a certain intensity is then gradually added to the implicit representation of the first image to obtain a third image.

[0140] Exemplarily, the image encoder may be any encoder capable of mapping an image from a pixel space to an implicit space. For example, the image encoder may be a variational auto-encoder (VAE-Encoder).

[0141] Exemplarily, the random noise may be Gaussian noise or other regular noise.

[0142] Step 1021: The electronic device inputs the third image, the prompt word information and the mask image into a diffusion model to obtain a second image.

[0143] It should be noted that the specific implementation of step 1021 can refer to the relevant description of the above step 102, and this embodiment will not be repeated here.

[0144] Optionally, when the prompt word information is represented in the form of text, the electronic device may first extract features from the prompt word information to obtain prompt word features, and then input the third image, the prompt word features and the mask image into the diffusion model to obtain the second image.

[0145] Exemplarily, the electronic device may input the prompt word information into a text encoder or any feature extraction device capable of extracting features, so as to extract features of the prompt word information and obtain prompt word features.

[0146] It should be noted that the specific implementation of the electronic device inputting the third image, the prompt word features and the mask image into the diffusion model to obtain the second image can also refer to the relevant description of the above step 102, which will not be repeated in this embodiment.

[0147] The image generation method provided by the embodiment of the present application, since the diffusion model itself has the function of image optimization, and the above method uses prompt word information, and the prompt word information is related to the image generation task, and the image generation task is generated according to user needs, then the obtained second image is more in line with user needs, and the above method also uses a mask image in the image optimization process, so that the image optimization process pays more attention to the key area corresponding to the mask image in the first image, and since the mask image is a mask image of the reference image, and the reference image corresponds to the real physical world, the optimization of the first image is based on the optimization of the real physical world, which is more in line with the actual situation. Therefore, the second image obtained by this solution has better quality, is more in line with the real physical world, meets user needs, and improves the image quality of the image taken by the electronic device. That is, the above method uses the generative artificial intelligence technology of the diffusion model to optimize the image, and in the case of low-quality original images, very beautiful night sky photography photos can still be obtained, which improves the image quality of the starry sky image taken by the electronic device. In addition, in the process of generating high-quality pictures, it is ensured that the position of the stars is in line with the real physical world, rather than generated out of thin air.

[0148] Figure 7 It is a flow chart of the image generation method provided in an embodiment of the present application. Taking a user taking photos of the starry sky with a mobile phone as an example, the method may include the following steps 11 to 21.

[0149] Step 11. Take photos of the starry sky with your mobile phone.

[0150] Step 12: The mobile phone obtains its own positioning information.

[0151] Exemplarily, the GPS positioning capability of the mobile phone is used to obtain the real-time location of the mobile phone at the moment of taking the photo.

[0152] Step 13: The mobile phone obtains its own posture information.

[0153] Exemplarily, the gyroscope data of the mobile phone is used to obtain the posture information of the mobile phone at the moment of taking a photo, and to calculate the orientation and pitch angle of the mobile phone camera.

[0154] Exemplarily, the above-mentioned heading and pitch angle are respectively measured in the following two angles:

[0155] North-east or north-west angle: 0 to 180 degrees. 0 degrees is due north, 180 degrees is due south.

[0156] Pitch angle: 0~90 degrees: 0 degrees horizontally forward, 90 degrees vertically upward.

[0157] Step 14: The mobile phone determines the shooting angle according to the positioning information and the attitude information, and obtains a reference star map under the shooting angle.

[0158] Step 15: The mobile phone generates a mask image based on the reference star map.

[0159] Exemplarily, the mobile phone searches the star map according to the star map calculation formula to obtain the positions of all stars under the above shooting angle, and constructs mask images with different intensity values ​​according to different brightness levels of the stars.

[0160] Step 16: The mobile phone obtains the prompt word information input by the user.

[0161] Step 17: The mobile phone encodes and performs noise processing on the first image to obtain a noisy image, and performs feature extraction processing on the prompt word information to obtain prompt word features.

[0162] It should be noted that the above-mentioned image after adding noise is the third image in the above-mentioned embodiment.

[0163] Step 18. The mobile phone inputs the noisy image, the prompt word features and the mask image into the diffusion model, and performs linear transformation processing on the target image through the i-th cross-attention module among the N cross-attention modules in the diffusion model to obtain the i-th query vector Q, and performs linear transformation processing on the prompt word features to obtain the i-th key vector K and the i-th value vector V.

[0164] Step 19: The mobile phone calculates the i-th intermediate feature map through the i-th cross-attention module based on the i-th query vector Q, the i-th key vector K, the i-th value vector V and the mask image.

[0165] Step 20: After obtaining the Nth intermediate feature map, the mobile phone iterates the feature processing M times, and uses the last intermediate feature map obtained after M iterations as the first implicit feature map.

[0166] Step 21: The mobile phone decodes the first implicit feature map to obtain a second image.

[0167] For example, see Figure 8 , the mobile phone takes a starry sky image and obtains the position information (GPS positioning information) and attitude information. According to the position information and attitude information, the star map under the corresponding shooting angle is obtained from the star map database, and the star map is masked to obtain a mask image. The starry sky image, the mask image and the prompt word information are input into the image generation model to obtain the output image.

[0168] It should be noted that the specific implementation process of the above steps 11 to 21 can refer to the relevant description of the above embodiment. To avoid repetition, this embodiment will not be repeated here.

[0169] Fig. 9It is a flow chart of the image generation method provided in an embodiment of the present application. Taking a user taking photos of the starry sky with a mobile phone as an example, the method may include the following steps 31 to 43.

[0170] Step 31. Take photos of the starry sky with your mobile phone.

[0171] Step 32: The mobile phone obtains its own positioning information.

[0172] Step 33: The mobile phone obtains its own posture information.

[0173] Step 34: The mobile phone determines the shooting angle according to the positioning information and the attitude information, and obtains a reference star map under the shooting angle.

[0174] Step 35: The mobile phone generates a mask image based on the reference star map.

[0175] Step 36: The mobile phone obtains the prompt word information input by the user.

[0176] Step 37: The mobile phone encodes and performs noise processing on the first image to obtain a noisy image, and performs feature extraction on the prompt word information to obtain prompt word features.

[0177] It should be noted that the above-mentioned image after adding noise is the third image in the above-mentioned embodiment.

[0178] Step 38: The mobile phone performs feature processing on the noisy image and prompt features through N cross attention modules in the diffusion model to obtain N first feature maps.

[0179] Step 39: The mobile phone calculates a first difference value based on the N first feature maps and the mask image through a difference calculation function.

[0180] Step 40: When the first difference value is less than or equal to the first threshold, a second image is obtained based on the last feature map in the N first feature maps, the prompt word feature and the mask image through N cross attention modules in the diffusion model.

[0181] Step 41: When the first difference value is greater than the first threshold, the mobile phone iteratively updates the feature map parameters in the difference calculation function until the difference value obtained by the difference calculation function is less than or equal to the first threshold.

[0182] Step 42: The mobile phone iterates feature processing Y times through the N cross attention modules in the diffusion model based on the iteratively updated feature map parameters and prompt word features, and uses the last feature map obtained after Y iterations as the second implicit feature map.

[0183] Step 43: The mobile phone decodes the second implicit feature map to obtain a second image.

[0184] It should be noted that the specific implementation process of the above steps 31 to 43 can refer to the relevant description of the above embodiment. To avoid repetition, this embodiment will not be repeated here.

[0185] It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not impose any restrictions on this.

[0186] The image generation method provided in the embodiment of the present application can be executed by an image generation device. In the embodiment of the present application, the image generation device provided in the embodiment of the present application is described by taking the image generation method executed by the image generation device as an example.

[0187] Fig.10 It is a structural schematic diagram of an image generating device provided in an embodiment of the present application. The image generating device includes: an acquisition module 71 and a processing module 72.

[0188] The acquisition module 71 is used to acquire prompt word information and a mask image corresponding to the first image. The prompt word information is description information of the image generation task corresponding to the first image, and the mask image is a mask image of a reference image under the shooting angle of the first image.

[0189] The processing module 72 is used to input the first image, the prompt word information and the mask image into the diffusion model to obtain the second image.

[0190] Optionally, in the embodiment of the present application, the processing module 72 is specifically used to:

[0191] The first image, the prompt word information obtained by the acquisition module 71, and the mask image are feature processed by the N cross attention modules in the diffusion model to obtain a first implicit feature map, where the first implicit feature map represents the position of the feature pixel points in the first image in the implicit space, and N is an integer greater than 1;

[0192] The first implicit feature map is decoded to obtain a second image.

[0193] Optionally, in the embodiment of the present application, the processing module 72 is specifically used to:

[0194] Through the i-th cross-attention module among the N cross-attention modules, the target image is linearly transformed to obtain the i-th query vector Q, and the prompt word information is linearly transformed to obtain the i-th key vector K and the i-th value vector V;

[0195] By using the i-th cross attention module, based on the i-th query vector Q, the i-th key vector K, the i-th value vector V and the mask image, an i-th intermediate feature map is calculated;

[0196] After obtaining the Nth intermediate feature map, the feature processing is iterated M times, and the last intermediate feature map obtained after iterating M times is used as the first implicit feature map;

[0197] Among them, i=1, the target image is the first image; i=2,…,N, the target image is the i-1th intermediate feature map; M is an integer greater than 1.

[0198] Optionally, in the embodiment of the present application, the processing module 72 is specifically used to:

[0199] Calculate the product of the i-th query vector Q and the i-th key vector K through the i-th cross attention module to obtain a first result;

[0200] Through the i-th cross attention module, the first result is normalized to obtain a second result;

[0201] Through the i-th cross attention module, the product of the second result, the mask image and the i-th value vector V is calculated to obtain the i-th intermediate feature map.

[0202] Optionally, in the embodiment of the present application, the processing module 72 is specifically used to:

[0203] Through the L cross attention modules in the diffusion model, feature processing is performed on the first image and the prompt word information obtained by the acquisition module 71 to obtain L first feature maps, each cross attention module corresponds to a first feature map, and L is an integer greater than 1;

[0204] By using a difference calculation function, based on the L first feature maps and the mask image, a first difference value is calculated, wherein the first difference value represents the difference between the L first feature maps and the mask image;

[0205] When the first difference value is greater than the first threshold, iteratively updating the characteristic map parameters in the difference calculation function until the difference value obtained by the difference calculation function is less than or equal to the first threshold;

[0206] Through the L cross attention modules in the diffusion model, based on the iteratively updated feature map parameters and the prompt word information, feature processing is iterated Y times, and the last feature map obtained after Y iterations is used as the second implicit feature map, where the second implicit feature map represents the position of the feature pixel points in the first image in the implicit space, and Y is an integer greater than 1;

[0207] The second implicit feature map is decoded to obtain a second image.

[0208] Optionally, in the embodiment of the present application, the processing module 72 is further configured to:

[0209] When the first difference value is less than or equal to the first threshold, a second image is obtained through the L cross attention modules in the diffusion model based on the last feature map in the L first feature maps, the prompt word information and the mask image.

[0210] Optionally, in an embodiment of the present application, the first image is a captured starry sky image, and the reference image is a star map.

[0211] Optionally, in the embodiment of the present application, the processing module 72 is further used to encode and denoise the first image to obtain a third image; and input the third image, prompt word information and mask image into the diffusion model to obtain a second image.

[0212] The image generation device provided in the embodiment of the present application has the function of image optimization because the diffusion model itself has the function of image optimization, and the above method uses prompt word information, and the prompt word information is related to the image generation task, and the image generation task is generated according to user needs, so the obtained second image is more in line with user needs. The above image generation device also uses a mask image in the image optimization process, so that the image optimization process pays more attention to the key areas corresponding to the mask image in the first image, and because the mask image is a mask image of the reference image, and the reference image corresponds to the real physical world, the optimization of the first image is based on the optimization of the real physical world, which is more in line with the actual situation. Therefore, the second image obtained by the above image generation device has better quality, is more in line with the real physical world, meets user needs, and improves the image quality of the image taken by the image generation device.

[0213] The image generating device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device, an augmented reality / virtual reality device, a robot, a wearable device, a super mobile personal computer, a netbook or a personal digital assistant, etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0214] The image generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0215] The image generating device provided in the embodiment of the present application can implement each process implemented by each embodiment of the above-mentioned image generating method, and will not be described again here to avoid repetition.

[0216] Alternatively, if Fig.11 As shown, an embodiment of the present application further provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores programs or instructions that can be executed on the processor 901, and when the program or instructions are executed by the processor 901, the various steps of the above-mentioned image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described again here.

[0217] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0218] Fig.12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0219] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0220] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0221] The user input unit 1007 is used to obtain prompt word information corresponding to the first image, where the prompt word information is description information of the image generation task corresponding to the first image.

[0222] The processor 1010 is used to obtain a mask image corresponding to the first image, where the mask image is a mask image of a reference image under the shooting angle of the first image; and input the first image, the prompt word information and the mask image into a diffusion model to obtain a second image.

[0223] Optionally, in an embodiment of the present application, the processor 1010 is specifically used to perform feature processing on the first image, the prompt word information obtained by the user input unit 1007 and the mask image through the N cross-attention modules in the diffusion model to obtain a first implicit feature map, where the first implicit feature map represents the position of feature pixel points in the first image in the implicit space, and N is an integer greater than 1; and, to perform decoding processing on the first implicit feature map to obtain a second image.

[0224] Optionally, in an embodiment of the present application, the processor 1010 is specifically used to perform linear transformation processing on the target image through the i-th cross-attention module among the N cross-attention modules to obtain the i-th query vector Q, and perform linear transformation processing on the prompt word information to obtain the i-th key vector K and the i-th value vector V; and, through the i-th cross-attention module, based on the i-th query vector Q, the i-th key vector K, the i-th value vector V and the mask image, calculate the i-th intermediate feature map; and, after obtaining the N-th intermediate feature map, iterate the feature processing M times, and use the last intermediate feature map obtained after the M iterations as the first implicit feature map; wherein, i=1, the target image is the first image; i=2,…,N, the target image is the i-1-th intermediate feature map, and M is an integer greater than 1.

[0225] Optionally, in an embodiment of the present application, the processor 1010 is specifically used to calculate the product of the i-th query vector Q and the i-th key vector K through the i-th cross-attention module to obtain a first result; and, through the i-th cross-attention module, normalize the first result to obtain a second result; and, through the i-th cross-attention module, calculate the product of the second result, the mask image and the i-th value vector V to obtain the i-th intermediate feature map.

[0226] Optionally, in an embodiment of the present application, the processor 1010 is specifically used to perform feature processing on the first image and the prompt word information through the L cross-attention modules in the diffusion model to obtain L first feature maps, each cross-attention module corresponds to a first feature map; and, through a difference calculation function, based on the L first feature maps and the mask image, calculate a first difference value, the first difference value represents the difference between the L first feature maps and the mask image; and, when the first difference value is greater than a first threshold, iteratively update the feature map parameters in the difference calculation function until the difference value obtained by the difference calculation function is less than or equal to the first threshold; and, through the L cross-attention modules in the diffusion model, based on the iteratively updated feature map parameters and the prompt word information, iterate the feature processing Y times, and use the last feature map obtained after Y iterations as the second implicit feature map, the second implicit feature map represents the position of the feature pixel point in the first image in the implicit space, Y is an integer greater than 1; and, decode the second implicit feature map to obtain a second image.

[0227] Optionally, in an embodiment of the present application, the processor 1010 is also used to obtain a second image based on the last feature map of the L first feature maps, the prompt word information and the mask image through the L cross-attention modules in the diffusion model when the first difference value is less than or equal to the first threshold.

[0228] Optionally, in an embodiment of the present application, the first image is a captured starry sky image, and the reference image is a star map.

[0229] Optionally, in an embodiment of the present application, the processor 1010 is further configured to encode and denoise the first image to obtain a third image; and input the third image, the prompt word information and the mask image into a diffusion model to obtain a second image.

[0230] The electronic device provided by the embodiment of the present application has the function of image optimization because the diffusion model itself has the function of image optimization, and the above method uses prompt word information, and the prompt word information is related to the image generation task, and the image generation task is generated according to user needs, so the second image obtained is more in line with user needs. The above method also uses a mask image in the image optimization process, so that the image optimization process pays more attention to the key areas corresponding to the mask image in the first image, and because the mask image is a mask image of the reference image, and the reference image corresponds to the real physical world, the optimization of the first image is based on the optimization of the real physical world, which is more in line with the actual situation. Therefore, the second image obtained by the electronic device has better quality, is more in line with the real physical world, meets user needs, and improves the image quality of images taken by the electronic device.

[0231] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0232] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0233] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0234] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0235] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0236] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0237] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0238] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0239] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0240] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0241] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire prompt word information and a mask image corresponding to the first image, wherein the prompt word information is description information of the image generation task corresponding to the first image, and the mask image is a mask image of a reference image under the shooting angle of the first image; The first image, the prompt word information and the mask image are input into a diffusion model to obtain a second image.

2. The method according to claim 1, characterized in that The step of inputting the first image, the prompt word information and the mask image into a diffusion model to obtain a second image comprises: Performing feature processing on the first image, the prompt word information, and the mask image through the N cross attention modules in the diffusion model to obtain a first implicit feature map, where the first implicit feature map represents the position of feature pixel points in the first image in an implicit space, and N is an integer greater than 1; The first implicit feature map is decoded to obtain the second image.

3. The method according to claim 2, characterized in that The first image, the prompt word information and the mask image are subjected to feature processing by the N cross attention modules in the diffusion model to obtain a first implicit feature map, including: Through the i-th cross-attention module among the N cross-attention modules, a linear transformation process is performed on the target image to obtain an i-th query vector Q, and a linear transformation process is performed on the prompt word information to obtain an i-th key vector K and an i-th value vector V; By means of the i-th cross attention module, based on the i-th query vector Q, the i-th key vector K, the i-th value vector V and the mask image, an i-th intermediate feature map is calculated; After obtaining the Nth intermediate feature map, iterate the feature processing M times, and use the last intermediate feature map obtained after iterating the M times as the first implicit feature map; Among them, i=1, the target image is the first image; i=2,…,N, the target image is the i-1th intermediate feature map; M is an integer greater than 1.

4. The method according to claim 3, characterized in that: The step of calculating, by the i-th cross attention module, an i-th intermediate feature map based on the i-th query vector Q, the i-th key vector K, the i-th value vector V and the mask image comprises: Calculate the product of the i-th query vector Q and the i-th key vector K through the i-th cross attention module to obtain a first result; Normalizing the first result through the i-th cross attention module to obtain a second result; Through the i-th cross-attention module, the product of the second result, the mask image and the i-th value vector V is calculated to obtain the i-th intermediate feature map.

5. The method according to claim 1, characterized in that: The step of inputting the first image, the prompt word information and the mask image into a diffusion model to obtain a second image comprises: Performing feature processing on the first image and the prompt word information through L cross attention modules in the diffusion model to obtain L first feature maps, each cross attention module corresponds to a first feature map, and L is an integer greater than 1; By using a difference calculation function, based on the L first feature maps and the mask image, a first difference value is calculated, where the first difference value represents a difference between the L first feature maps and the mask image; When the first difference value is greater than a first threshold, iteratively updating the characteristic map parameters in the difference calculation function until the difference value obtained by the difference calculation function is less than or equal to the first threshold; Through the L cross attention modules in the diffusion model, based on the iteratively updated feature map parameters and the prompt word information, iterate the feature processing for Y times, and use the last feature map obtained after the Y iterations as the second implicit feature map, where the second implicit feature map represents the position of the feature pixel points in the first image in the implicit space, and Y is an integer greater than 1; The second implicit feature map is decoded to obtain the second image.

6. The method according to claim 5, characterized in that The method further comprises: When the first difference value is less than or equal to the first threshold, the second image is obtained based on the last feature map of the L first feature maps, the prompt word information and the mask image through L cross attention modules in the diffusion model.

7. The method according to any one of claims 1 to 6, characterized in that: The first image is a captured starry sky image, and the reference image is a star map.

8. The method according to any one of claims 1 to 6, characterized in that: Before inputting the first image, the prompt word information and the mask image into a diffusion model to obtain a second image, the method further includes: encoding and adding noise to the first image to obtain a third image; Inputting the first image, the prompt word information and the mask image into a diffusion model to obtain a second image includes: The third image, the prompt word information and the mask image are input into a diffusion model to obtain the second image.

9. An image generating device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to acquire prompt word information and a mask image corresponding to the first image, wherein the prompt word information is description information of the image generation task corresponding to the first image, and the mask image is a mask image of a reference image under the shooting angle of the first image; The processing module is used to input the first image, the prompt word information obtained by the obtaining module and the mask image into a diffusion model to obtain a second image.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image generation method according to any one of claims 1 to 8 are implemented.

11. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the image generation method according to any one of claims 1 to 8 are implemented.

Images