Image generation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides an image generation method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining an expected text used for describing an expectation corresponding to generation of a target image, wherein the expected text comprises a text used for describing an action posture of a target object; obtaining a reference image of the expected text, wherein the action posture of a reference object in the reference image is matched with the action posture described by the text; when the reference object is different from the target object, replacing the face of the reference object in the reference image with the face of the target object to obtain a target reference image; and in combination with the expected text and the target reference image, predicting to obtain a target image meeting the expectation. Through the method and the device, the information fitting degree between the expected text and the target image can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image generation method, device, electronic device, storage medium and program product. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies.

[0003] In the related art, in order to generate a target image corresponding to the expected text, the corresponding target image is usually predicted based on the expected text features corresponding to the expected text. In this way, the information fit between the expected text and the target image often depends on the accuracy of the prediction. Usually, the accuracy of the prediction cannot be 100% accurate, which results in a low information fit between the expected text and the target image. Summary of the invention

[0004] The embodiments of the present application provide an image generation method, device, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the information fit between the expected text and the target image.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides an image generation method, including:

[0007] Acquire an expected text for describing the expectation corresponding to generating the target image, wherein the expected text includes text for describing the action posture of the target object;

[0008] Acquire a reference image of the expected text, wherein the motion posture of a reference object in the reference image matches the motion posture described in the text;

[0009] When the reference object is different from the target object, replacing the face of the reference object in the reference image with the face of the target object to obtain a target reference image;

[0010] The expected text and the target reference image are combined to predict a target image that meets the expectation.

[0011] The present application provides an image generating device, including:

[0012] A text acquisition module, used to acquire an expected text for describing the expected corresponding to the generated target image, wherein the expected text includes a text for describing the action posture of the target object;

[0013] An image acquisition module, used to acquire a reference image of the expected text, wherein the action posture of the reference object in the reference image matches the action posture described in the text;

[0014] a replacement module, configured to replace the face of the reference object in the reference image with the face of the target object to obtain a target reference image when the reference object is different from the target object;

[0015] The prediction module is used to combine the expected text and the target reference image to predict a target image that meets the expectation.

[0016] In the above scheme, the above-mentioned image acquisition module is used to obtain the mapping relationship between multiple candidate action postures and corresponding candidate reference images, wherein the candidate reference images include reference objects corresponding to the candidate action postures; the action posture of the target object is matched with each of the candidate action postures, and the matching degree between the action posture of the target object and each of the candidate action postures is obtained, and the candidate action posture corresponding to the highest matching degree is determined as the target action posture matching the action posture of the target object; based on the mapping relationship, the candidate reference image corresponding to the target action posture is determined as the reference image.

[0017] In the above scheme, the above replacement module is used to obtain the facial image of the target object and the image size of the facial image; determine the facial angle of the reference object in the reference image and the facial size of the reference object in the reference image; adjust the image size of the facial image to the facial size to obtain the resized image of the facial image; obtain the facial angle of the target object in the resized image, and adjust the facial angle of the target object in the resized image to the facial angle of the reference object to obtain the target map; replace the image content of the area where the face of the reference object in the reference image is located with the target map to obtain the target reference image.

[0018] In the above scheme, the above prediction module is also used to obtain the target text features of the expected text, and the target text features are used to indicate at least one historical action posture corresponding to the target object; perform image feature extraction on the target reference image to obtain the image features of the target reference image; perform feature fusion on the image features and the target text features at least once to obtain fusion features corresponding to each feature fusion; combine the target text features and each fusion feature to predict a target image that meets the expectations.

[0019] In the above scheme, the above prediction module is also used to extract text features of the expected text to obtain expected text features of the expected text, wherein the expected text features include identification text features for identifying the target object; obtain updated text features of the target object, wherein the updated text features are used to indicate at least one historical action posture corresponding to the target object; replace the identification text features in the expected text features with the updated text features to obtain the target text features.

[0020] In the above scheme, the above prediction module is also used to obtain the mapping relationship between multiple candidate objects and corresponding text features, and the multiple candidate objects include the target object; query the target mapping relationship including the target object from the mapping relationship, and determine the text feature in the target mapping relationship as the updated text feature.

[0021] In the above scheme, the above prediction module is also used to perform feature fusion on the image feature and the target text feature when the number of feature fusions is multiple times, so as to obtain the first fusion feature; perform feature fusion on the first fusion feature and the target text feature at least once, so as to obtain the fusion features corresponding to each feature fusion.

[0022] In the above scheme, the above prediction module is also used to traverse i to perform the following processing: perform feature fusion on the i-th fusion feature and the target text feature to obtain the i+1-th fusion feature; wherein 1≤i≤N-1, N is the maximum number of feature fusions.

[0023] In the above scheme, the above prediction module is also used to obtain an image prediction network, which includes a downsampling network, an upsampling network and a prediction network; calling the downsampling network to downsample the target text features to obtain reference features, and fusing the reference features with the fused features obtained by the last feature fusion to obtain target fused features; calling the upsampling network to upsample the target fused features to obtain target features; calling the prediction network to predict a target image that meets the expectations based on the target features.

[0024] In the above scheme, the downsampling network includes at least one downsampling layer, and the number of the downsampling layers is the same as the number of times the feature is fused. The above prediction module is also used to call the first downsampling layer when the number of times the feature is fused is multiple, downsample the target text feature, and obtain the first reference feature; traverse j to perform the following processing: fuse the jth reference feature with the fusion feature corresponding to the jth feature fusion to obtain the jth target fusion feature; call the jth downsampling layer, downsample the jth target fusion feature, and obtain the j+1th reference feature; wherein 1≤j≤M-1, M is the number of the downsampling layers in the downsampling network; determine the Mth reference feature as the reference feature.

[0025] In the above scheme, the above prediction module is also used to obtain an initial image prediction network, a target image sample, and an expected text sample for describing the expected value corresponding to the target image sample; calling the initial image prediction network, predicting a first predicted image based on the expected text sample, and determining a first loss value based on the first predicted image and the target image sample; obtaining a reference image sample of the expected text sample, and calling the initial image prediction network, combining the reference image sample and the expected text sample, predicting a second predicted image, and determining a second loss value based on the second predicted image and the target image sample; training the initial image prediction network based on the first loss value and the second loss value to obtain the image prediction network.

[0026] An embodiment of the present application provides an electronic device, including:

[0027] A memory for storing computer executable instructions or computer programs;

[0028] The processor is used to implement the image generation method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0029] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for causing a processor to execute and implement the image generation method provided in the embodiment of the present application.

[0030] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction or the computer program from the computer-readable storage medium, and the processor executes the computer executable instruction or the computer program, so that the electronic device performs the image generation method described in the embodiment of the present application.

[0031] The embodiments of the present application have the following beneficial effects:

[0032] By obtaining the expected text used to describe the expectation corresponding to the generated target image and the reference image of the expected text, when the reference object in the reference image is different from the target object in the expected text, the face of the reference object in the reference image is replaced with the face of the target object to obtain the target reference image, and the target image that meets the expectation is predicted by combining the expected text and the target reference image. In this way, by obtaining a reference image that matches the action posture described in the text, and replacing the face of the reference object in the reference image with the face of the target object, the face of the object in the obtained target reference image is the face of the target object, and the action posture also matches the action posture described in the text, so that the target reference image can basically meet the action posture of the target object described in the expected text, and thus combined with the expected text and the target reference image, the predicted target image that meets the expectation can better fit the action posture of the target object described in the expected text, thereby effectively improving the information fit between the expected text and the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic diagram of the architecture of a system for image generation provided in an embodiment of the present application;

[0034] Figure 2 is a schematic diagram of the structure of an electronic device for image generation provided in an embodiment of the present application;

[0035] Figures 3 to 6 It is a flowchart of the image generation method provided in the embodiment of the present application;

[0036] Figures 7 to 11 It is a schematic diagram of the principle of the image generation method provided in the embodiment of the present application;

[0037] Fig.12 It is a schematic diagram of the principle of the line drawing image provided by the embodiment of the present application;

[0038] Fig.13 It is a schematic diagram of the principle of a facial image provided by an embodiment of the present application;

[0039] Fig.14 It is a schematic diagram of the principle of the image generation method provided in the embodiment of the present application;

[0040] Fig.15 It is a flowchart of the image generation method provided in the embodiment of the present application;

[0041] Fig.16 It is a schematic diagram of the principle of the image generation method provided in the embodiment of the present application;

[0042] Fig.17It is a schematic diagram of the effect of the rotation angle provided in the embodiment of the present application;

[0043] Fig.18 It is a schematic diagram of the principle of face replacement provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0045] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0048] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0049] 1) Machine Learning (ML): It is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0050] 2) Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies.

[0051] 3) Convolutional Neural Networks (CNN): It is a type of feed forward neural network (FNN) that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input images according to their hierarchical structure.

[0052] 4) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0053] 5) Convolutional layer: Each convolutional layer in a convolutional neural network consists of several convolutional units, and the parameters of each convolutional unit are optimized through the back propagation algorithm. The purpose of the convolution operation is to extract different features of the input. The first convolutional layer may only extract some low-level features such as edges, lines, and corners. More layers of the network can iteratively extract more complex features from low-level features.

[0054] 6) Pooling layer: After the convolution layer performs feature extraction, the output feature map is passed to the pooling layer for feature selection and information filtering. The pooling layer contains a pre-set pooling function, which replaces the result of a single point in the feature map with the feature map statistics of its adjacent area. The pooling layer selects the pooling area in the same way as the convolution kernel scans the feature map, which is controlled by the pooling size, step size, and padding.

[0055] 7) Fully-Connected Layer: The fully connected layer in a convolutional neural network is equivalent to the hidden layer in a traditional feedforward neural network. The fully connected layer is located at the end of the hidden layer of the convolutional neural network and only transmits signals to other fully connected layers. The feature map loses its spatial topological structure in the fully connected layer, is expanded into a vector and passes through the activation function.

[0056] In the related art, in order to generate a target image corresponding to the expected text, the corresponding target image is usually predicted based on the expected text features corresponding to the expected text. In this way, the information fit between the expected text and the target image often depends on the accuracy of the prediction. Usually, the accuracy of the prediction cannot be 100% accurate, which results in a low information fit between the expected text and the target image.

[0057] The embodiments of the present application provide an image generation method, device, electronic device, computer-readable storage medium and computer program product, which can effectively improve the information fit between the desired text and the target image. The following describes an exemplary application of the image processing system provided by the embodiments of the present application.

[0058] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of a system 100 for image generation provided in an embodiment of the present application. A terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0059] The terminal 400 is used for the user to use the client 410 to display a target image on a graphical interface 410-1 (graphic interface 410-1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0060] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, a car terminal, etc., but is not limited thereto. The electronic device provided in the embodiment of the present application may be implemented as a terminal or as a server. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application.

[0061] In some embodiments, the server 200 obtains the expected text and the reference image from the terminal 400, and when the reference object and the target object are different, replaces the face of the reference object in the reference image with the face of the target object to obtain the target reference image, combines the expected text and the target reference image, predicts a target image that meets the expectations, and sends the target image to the terminal 400.

[0062] In other embodiments, the terminal 400 obtains the expected text and reference image from the server 200, and when the reference object and the target object are different, replaces the face of the reference object in the reference image with the face of the target object to obtain a target reference image, combines the expected text and the target reference image, predicts a target image that meets the expectations, displays the target image, and sends the target image to the server 200.

[0063] In other embodiments, the embodiments of the present application can be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data computing, storage, processing, and sharing.

[0064] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0065] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 for image generation provided in an embodiment of the present application, wherein: Figure 2 The electronic device 500 shown may be Figure 1 The server 200 or the terminal 400 in Figure 2 The electronic device 500 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0066] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0067] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0068] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0069] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0070] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0071] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).

[0072] In some embodiments, the image generation device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 The image generation device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a text acquisition module 4551, an image acquisition module 4552, a replacement module 4553, and a prediction module 4554. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0073] In other embodiments, the image processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the image processing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0074] In some embodiments, the terminal or server can implement the image generation method provided in the embodiment of the present application by running a computer program or a computer executable instruction. For example, the computer program can be a native program (e.g., a dedicated image generation program) or a software module in the operating system, for example, an image generation module that can be embedded in any program (such as a video client, an instant messaging client, an album program, an electronic map client, a navigation client); for example, it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run. In short, the above-mentioned computer program can be any form of application, module or plug-in.

[0075] The image generation method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the server or terminal provided in the embodiment of the present application.

[0076] See also Figure 3 , Figure 3 is a flow chart of the image generation method provided in the embodiment of the present application, which will be combined with Figure 3 Steps 101 to 104 are shown for illustration. The image generation method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following description will be made using the server alone as an example.

[0077] In step 101, an expected text for describing the expectation corresponding to generating a target image is obtained.

[0078] In some embodiments, the desired text includes text for describing the action gesture of the target object.

[0079] As an example, see Figure 8 , Figure 8 is a schematic diagram of the principle of the image generation method provided in the embodiment of the present application. The expected text can be Figure 8 The text shown in: Zhang San stood and looked up at the sky.

[0080] In some embodiments, the expected text also includes text indicating an identifier of the target object, and the identifier of the target object may be a name of the target object.

[0081] In step 102, a reference image of the desired text is obtained.

[0082] In some embodiments, the motion pose of the reference object in the reference image is matched with the motion pose described in the text.

[0083] As an example, see Figure 8 , the reference image of the expected text can be, Figure 8 In the reference picture shown, Li Si is looking up at the sky.

[0084] In some embodiments, the action posture of the reference object in the reference image matches the action posture of the text description, which means that the matching degree between the action posture of the reference object in the reference image and the action posture of the text description is greater than a matching degree threshold. When the matching degree between the action posture of the reference object in the reference image and the action posture of the text description is 1, the action posture of the reference object in the reference image is the same as the action posture of the text description.

[0085] As an example, see Figure 8 , the action posture of the reference object in the reference image is: looking up at the sky, and the action posture described in the text is: standing and looking up at the sky. The matching degree between the action posture of the reference object in the reference image and the action posture described in the text can be 0.9. When the matching degree threshold is 0.6, the matching degree between the action posture of the reference object in the reference image and the action posture described in the text is greater than the matching degree threshold.

[0086] In this way, since the action posture of the reference object in the reference image matches the action posture described in the text, the reference image provides strong image data support for the expected text, and can provide a matching image description for the expected text through the reference image. By combining the expected text and the reference image, a target image that meets the expectations is predicted, and the obtained predicted image can be more consistent with the action posture described in the text in terms of action posture, effectively improving the information fit between the expected text and the target image.

[0087] In some embodiments, see Figure 4 , Figure 4 It is a flowchart of the image generation method provided in an embodiment of the present application. Figure 3Step 102 shown may be performed by executing Figure 4 Steps 1021 to 1024 are implemented as shown.

[0088] In step 1021, mapping relationships between a plurality of candidate action postures and corresponding candidate reference images are obtained.

[0089] In some embodiments, the candidate reference images include reference objects of corresponding candidate action poses.

[0090] As an example, the mapping relationship between a plurality of candidate action postures and corresponding candidate reference images can be schematically shown in Table 1 below.

[0091] Table 1 Mapping relationship between multiple candidate action postures and corresponding candidate reference images

[0092] Candidate Action Postures Candidate reference images Candidate action pose A Candidate reference image A Candidate action pose B Candidate reference image B Candidate action pose C Candidate reference image C

[0093] Among them, candidate reference image A includes a reference object of the corresponding candidate action posture A, candidate reference image B includes a reference object of the corresponding candidate action posture B, and candidate reference image C includes a reference object of the corresponding candidate action posture C.

[0094] In step 1022, the action posture of the target object is matched with each candidate action posture respectively to obtain the matching degree between the action posture of the target object and each candidate action posture.

[0095] In some embodiments, the above step 1022 can be implemented as follows: perform the following processing for each candidate action gesture: match the action gesture of the target object with the candidate action gesture to obtain the matching degree between the action gesture of the target object and the candidate action gesture.

[0096] In some embodiments, the above-mentioned matching of the target object's action posture and the candidate action posture can be achieved by calculating the similarity between the target object's action posture and the candidate action posture (i.e., the matching degree described above), and the above-mentioned similarity can be in the form of cosine similarity, Jaccard similarity coefficient, or other similarity forms.

[0097] In step 1023, the candidate action posture corresponding to the highest matching degree is determined as the target action posture that matches the action posture of the target object.

[0098] As an example, the matching degree corresponding to candidate action posture A is 0.5, the matching degree corresponding to candidate action posture B is 0.6, and the matching degree corresponding to candidate action posture C is 0.7. Candidate action posture C corresponding to the highest matching degree 0.7 is determined as the target action posture that matches the action posture of the target object.

[0099] In step 1024, based on the mapping relationship, the candidate reference image corresponding to the target action posture is determined as the reference image.

[0100] As an example, referring to Table 1 above, when the target action posture is the candidate action posture C, the candidate reference image C corresponding to the candidate action posture C is determined as the reference image.

[0101] In this way, by matching the action posture of the target object with each candidate action posture respectively, the candidate action posture corresponding to the highest matching degree is determined as the target action posture that matches the action posture of the target object. Since the candidate reference image includes the reference object of the corresponding candidate action posture, by determining the candidate reference image corresponding to the target action posture as the reference image, the action posture of the corresponding reference object included in the reference image has a high similarity with the action posture of the target object. The reference image provides strong image data support for the expected text, and can provide a matching image description for the expected text through the reference image. By combining the expected text and the reference image, the target image that meets the expectation is predicted, so that the obtained predicted image can be more consistent with the action posture described in the text in terms of action posture, effectively improving the information fit between the expected text and the target image.

[0102] In step 103, when the reference object is different from the target object, the face of the reference object in the reference image is replaced with the face of the target object to obtain a target reference image.

[0103] In some embodiments, when the reference image is identical to the target object, it indicates that the facial features of the reference object in the reference image are identical to the facial features of the target object. Therefore, there is no need to replace the face of the reference object in the reference image with the face of the target object, and the reference image can be directly determined as the target reference image.

[0104] In some embodiments, see Figure 5 , Figure 5 It is a flowchart of the image generation method provided in an embodiment of the present application. Figure 3 Step 103 shown may be performed by executing Figure 5 Steps 1031 to 1035 are implemented as shown.

[0105] In step 1031, a facial image of the target object and an image size of the facial image are obtained.

[0106] In some embodiments, the above-mentioned acquisition of the facial image of the target object can be achieved by: obtaining the mapping relationship between multiple candidate objects and the corresponding facial images, determining the mapping relationship including the target object as the target mapping relationship, and determining the facial image in the target mapping relationship as the facial image of the target object.

[0107] As an example, see Figure 7 , Figure 7 It is a schematic diagram of the principle of the image generation method provided in an embodiment of the present application, which obtains a facial image 71 of a target object and an image size of the facial image 71 (for example, the image size is 72×72).

[0108] In step 1032, the facial angle of the reference object in the reference image and the facial size of the reference object in the reference image are determined.

[0109] In some embodiments, the facial angle of the reference object in the reference image may refer to the orientation angle of the face of the reference object in the image reference coordinate system of the reference image. The facial angle of the reference object in the reference image is used to indicate the orientation of the face of the reference object in the reference image. The facial angle of the reference object in the reference image has different values ​​corresponding to different angles in different coordinate systems.

[0110] In step 1033, the image size of the facial image is adjusted to the facial size to obtain a size-adjusted image of the facial image.

[0111] In some embodiments, the above step 1033 can be implemented as follows: compare the facial size of the reference object in the reference image with the image size of the facial image of the target object to obtain a comparison result; when the comparison result indicates that the image size of the facial image is smaller than the facial size, enlarge the facial image to obtain a size-adjusted image of the facial image; when the comparison result indicates that the image size of the facial image is larger than the facial size, reduce the facial image to obtain a size-adjusted image of the facial image; when the comparison result indicates that the image size of the facial image is equal to the facial size, determine the facial image as a size-adjusted image. The image size of the size-adjusted image is equal to the facial size.

[0112] As an example, see Figure 7 , the image size of the target object's facial image 73 is adjusted to the facial size, and a size-adjusted image 74 of the facial image is obtained.

[0113] In step 1034, the facial angle of the target object in the resized image is obtained, and the facial angle of the target object in the resized image is adjusted to the facial angle of the reference object to obtain a target map.

[0114] In some embodiments, the facial angle of the target object in the resized image is adjusted to the facial angle of the reference object, so that the facial angle of the target object in the obtained target map can be consistent with the facial angle of the reference image.

[0115] In some embodiments, the execution order of the above steps 1033 to 1034 can also be to perform angle adjustment first and then size adjustment. Specifically, the facial angle of the target object in the facial image can be obtained, and the facial angle of the target object in the facial image can be adjusted to the facial angle of the reference object to obtain an angle-adjusted image, and then the image size of the angle-adjusted image can be adjusted to the facial size to obtain a target map.

[0116] As an example, see Figure 7 , obtain the facial angle of the target object in the facial image 71, and adjust the facial angle of the target object in the facial image 71 to the facial angle of the reference object to obtain an angle-adjusted image 72, detect a new face position 73 from the angle-adjusted image 72, and then adjust the image size of the angle-adjusted image 73 to the facial size to obtain a target map 74.

[0117] In step 1035, the image content of the area where the face of the reference object is located in the reference image is replaced with the target map to obtain a target reference image.

[0118] As an example, see Figure 7 , the image content of the area 70 where the face of the reference object in the reference image is located is replaced with the target map 74 to obtain a target reference image.

[0119] In this way, the target reference image is obtained by replacing the face of the reference object in the reference image with the face of the target object, so that the action posture of the corresponding reference object included in the target reference image has a high degree of similarity with the action posture of the target object, and the facial features of the corresponding reference image included in the target reference image are exactly the same as the facial features of the target object. The target reference image can provide a matching image description for the expected text, and by combining the expected text and the target reference image to predict the expected target image, the obtained predicted image can be more consistent with the action posture and facial features described in the text in terms of action posture, effectively improving the information fit between the expected text and the target image.

[0120] In step 104, the expected text and the target reference image are combined to predict a target image that meets the expectations.

[0121] In some embodiments, see Figure 6 , Figure 6 It is a flowchart of the image generation method provided in an embodiment of the present application. Figure 3 Step 104 shown may be performed by executing Figure 6 Steps 1041 to 1044 are shown to be implemented.

[0122] In step 1041, target text features of the expected text are obtained.

[0123] In some embodiments, the target text feature is used to indicate at least one historical action gesture corresponding to the target object.

[0124] In some embodiments, the above step 1041 can be implemented as follows: extract text features from the expected text to obtain expected text features of the expected text; obtain updated text features of the target object; replace the identification text features in the expected text features with updated text features to obtain target text features.

[0125] In some embodiments, the above-mentioned text feature extraction of the expected text to obtain the expected text features of the expected text can be achieved by obtaining a text encoding network, calling the text encoding network, performing text feature extraction on the expected text, and obtaining the expected text features of the expected text.

[0126] In some embodiments, the expected text feature includes an identification text feature for identifying the target object and an update text feature for indicating at least one historical action gesture corresponding to the target object.

[0127] In some embodiments, the above-mentioned acquisition of updated text features of the target object can be achieved in the following manner: obtaining mapping relationships between multiple candidate objects and corresponding text features, wherein the multiple candidate objects include the target object; querying a target mapping relationship including the target object from the mapping relationship, and determining the text features in the target mapping relationship as updated text features.

[0128] In other embodiments, the above-mentioned obtaining of updated text features of the target object may be implemented in the following manner: obtaining a feature prediction network, and calling the feature prediction network to predict updated text features based on expected text features.

[0129] As an example, see Figure 8 , Figure 8 is a schematic diagram of the principle of the image generation method provided in the embodiment of the present application. For the desired text (for example, Figure 8 The text shown in: Zhang San stood and looked up at the sky), the expected text ( Figure 8 The text shown in the figure: Zhang San stands and looks up at the sky) is subjected to text feature extraction to obtain expected text features of the expected text, and mapping relationships between multiple candidate objects and corresponding text features are obtained, wherein the multiple candidate objects include a target object; a target mapping relationship including the target object is queried from the mapping relationship, and the text features in the target mapping relationship are determined as updated text features, and the identification text features in the expected text features are replaced with the updated text features to obtain the target text features.

[0130] In step 1042, image features are extracted from the target reference image to obtain image features of the target reference image.

[0131] In some embodiments, the above step 1042 can be implemented in the following manner: obtaining an image feature extraction network, calling the image feature extraction network, performing image feature extraction on the target reference image, and obtaining image features of the target reference image.

[0132] In some embodiments, the above step 1042 can be implemented as follows: extract lines from the target reference image to obtain a line graph corresponding to the target reference image; obtain an image feature extraction network, call the image feature extraction network, perform image feature extraction on the line graph of the target reference image, and obtain image features of the target reference image.

[0133] In some embodiments, since the line graph corresponding to the target reference image can effectively reduce redundant information in the target reference image and retain valid information in the target reference image, line extraction is performed on the target reference image to obtain a line graph corresponding to the target reference image, and feature extraction is performed on the line graph to obtain image features of the target reference image, which can effectively reduce information redundancy of the image features.

[0134] In step 1043, feature fusion is performed on the image features and the target text features at least once to obtain fusion features corresponding to each feature fusion.

[0135] In some embodiments, the above step 1043 can be implemented in the following manner: when the number of feature fusions is multiple times, the image features and the target text features are fused to obtain the first fused features; the first fused features and the target text features are fused at least once to obtain fused features corresponding to each feature fusion.

[0136] As an example, see Figure 8 The above step 1043 can be implemented as follows: calling the control network, performing at least one feature fusion on the image feature and the target text feature, and obtaining the fusion feature corresponding to each feature fusion. The above control network includes multiple fusion layers, for example, fusion 1 and fusion 2. Fusion 1 is called to perform feature fusion on the image feature and the target text feature to obtain the first fusion feature; fusion 2 is called to perform feature fusion on the first fusion feature and the target text feature to obtain the fusion feature corresponding to the feature fusion.

[0137] In some embodiments, the above step 1043 can be implemented in the following manner: when the number of feature fusions is one, the image features and the target text features are fused to obtain fused features.

[0138] In some embodiments, the first fusion feature and the target text feature are fused at least once to obtain the fusion features corresponding to each feature fusion, which can be achieved as follows: traverse i and perform the following processing: feature fuse the i-th fusion feature and the target text feature to obtain the i+1-th fusion feature.

[0139] In some embodiments, 1≤i≤N-1, where N is the maximum number of feature fusions.

[0140] As an example, when N=3, the first fused feature and the target text feature are fused to obtain the second fused feature, and the second fused feature and the target text feature are fused to obtain the third fused feature.

[0141] As an example, when N=4, the first fusion feature and the target text feature are fused to obtain the second fusion feature, the second fusion feature and the target text feature are fused to obtain the third fusion feature, and the third fusion feature and the target text feature are fused to obtain the fourth fusion feature.

[0142] In some embodiments, the number of feature fusions is positively correlated with the weight of the target text feature in the corresponding fused feature.

[0143] In step 1044, the target text features and the fused features are combined to predict a target image that meets expectations.

[0144] In some embodiments, the above step 1044 can be implemented as follows: obtain an image prediction network; call a downsampling network to downsample the target text features to obtain a reference feature, and fuse the reference feature with the fused feature obtained by the last feature fusion to obtain a target fused feature; call an upsampling network to upsample the target fused feature to obtain a target feature; call a prediction network to predict a target image that meets expectations based on the target feature.

[0145] In some embodiments, the image prediction network includes a downsampling network, an upsampling network, and a prediction network.

[0146] In some embodiments, the downsampling network includes at least one downsampling layer, and the number of downsampling layers is the same as the number of feature fusions.

[0147] As an example, see Figure 8 , call the downsampling network, downsample the target text features, obtain the reference features, and fuse the reference features with the last feature fusion to obtain the fusion features ( Figure 8The output of fusion 2 shown in the figure is fused to obtain the target fusion feature; the upsampling network is called to upsample the target fusion feature to obtain the target feature; the prediction network Y is called to predict the target image that meets the expectations based on the target feature.

[0148] As an example, see Figure 8 , Figure 8 The downsampling network shown consists of two downsampling layers, Figure 8 The upsampling network shown includes two upsampling layers, and the number of downsampling layers is the same as the number of feature fusions, that is, the number of downsampling layers is equal to the number of fusion layers (fusion 1 and fusion 2) in the control network.

[0149] In some embodiments, the above-mentioned calling of the downsampling network to downsample the target text features to obtain reference features can be achieved in the following way: when the number of feature fusions is multiple times, calling the first downsampling layer to downsample the target text features to obtain the first reference features; traversing j to perform the following processing: fusing the jth reference feature and the fusion feature corresponding to the jth feature fusion to obtain the jth target fusion feature; calling the jth downsampling layer to downsample the jth target fusion feature to obtain the j+1th reference feature; determining the Mth reference feature as the reference feature.

[0150] In some embodiments, 1≤j≤M-1, where M is the number of downsampling layers in the downsampling network.

[0151] As an example, see Figure 8 When M=2, the first downsampling layer is called to downsample the target text feature to obtain the first reference feature, and the first reference feature is fused with the fusion feature corresponding to the first feature fusion to obtain the first target fusion feature; the first downsampling layer is called to downsample the first target fusion feature to obtain the first reference feature; the first reference feature is fused with the fusion feature corresponding to the first feature fusion to obtain the first target fusion feature; the first downsampling layer is called to downsample the first target fusion feature to obtain the second reference feature.

[0152] In some embodiments, the above-mentioned acquisition of the image prediction network can be implemented in the following manner: acquiring an initial image prediction network, a target image sample, and an expected text sample for describing the expectation corresponding to the generated target image sample; calling the initial image prediction network, predicting a first predicted image based on the expected text sample, and determining a first loss value based on the first predicted image and the target image sample; acquiring a reference image sample of the expected text sample, and calling the initial image prediction network, predicting a second predicted image based on the reference image sample and the expected text sample, and determining a second loss value based on the second predicted image and the target image sample; training the initial image prediction network based on the first loss value and the second loss value to obtain an image prediction network.

[0153] As an example, the expression for the first loss value may be:

[0154]

[0155] Among them, L 1 Used to indicate the first loss value, y i is used to indicate the pixel value of the i-th pixel in the first predicted image, Used to indicate the pixel value of the corresponding pixel point in the target image sample.

[0156] As an example, the expression of the second loss value may be:

[0157]

[0158] Among them, L 2 Used to indicate the second loss value, x i is used to indicate the pixel value of the i-th pixel in the second predicted image, Used to indicate the pixel value of the corresponding pixel point in the target image sample.

[0159] In some embodiments, the above-mentioned training of the initial image prediction network based on the first loss value and the second loss value to obtain the image prediction network can be achieved as follows: weighted summing the first loss value and the second loss value to obtain the summed loss value, and training the initial image prediction network based on the summed loss value to obtain the image prediction network.

[0160] In some embodiments, the control network can be trained as follows: obtain an initial control network, a target image sample, and an expected text sample for describing the expectation corresponding to the target image sample; call the image prediction network, predict a first predicted image based on the expected text sample, and determine a first loss value based on the first predicted image and the target image sample; obtain a reference image sample of the expected text sample, and call the initial image prediction network, combine the reference image sample and the expected text sample, predict a second predicted image, and determine a second loss value based on the second predicted image and the target image sample; train the initial control network based on the first loss value and the second loss value to obtain a control network.

[0161] Thus, by obtaining the expected text used to describe the expectation corresponding to the generated target image and the reference image of the expected text, when the reference object in the reference image is different from the target object in the expected text, the face of the reference object in the reference image is replaced with the face of the target object to obtain the target reference image, and the target text and the target reference image are combined to predict the target image that meets the expectation. Thus, by obtaining a reference image that matches the action posture described in the text, and replacing the face of the reference object in the reference image with the face of the target object, the face of the object in the obtained target reference image is the face of the target object, and the action posture also matches the action posture described in the text, so that the target reference image can basically meet the action posture of the target object described in the expected text, and thus the expected text and the target reference image are combined to predict that the target image that meets the expectation can better fit the action posture of the target object described in the expected text, thereby effectively improving the information fit between the expected text and the target image.

[0162] The following is an explanation of an exemplary application of the embodiments of the present application in an actual image generation application scenario.

[0163] The creation of martial arts legends often requires the generation of illustrations for a character description. Since martial arts characters have some specific martial arts actions, such as practicing the Eighteen Dragon Subduing Palms on the top of a mountain, it is difficult to cover the various actions and character scene layouts in film and television dramas in conventional generation model training. In order to improve the generation efficiency and effect of specific actions in the creation of martial arts characters, the embodiment of the present application designs a generation model method based on the control of the line draft of the reference film and television character actions and the fine-tuning of specific characters. When creating a martial arts action on the generation model, the embodiment of the present application retrieves the target effect diagram according to the historical martial arts inventory, and replaces the character of the target effect diagram line draft as the action control of the generation model to obtain the final generation. The embodiment of the present application replaces the character line draft diagram, fine-tunes and binds the line draft character with the target character, so that the target character can be generated by inputting the line draft diagram character and the target character name into the model, and finally supports the application of martial arts legends in the application. In the application, the description text is first retrieved and analyzed. When there are martial arts inventory-related images, the character action line draft diagram is used for creation, otherwise the normal literary image is used for creation.

[0164] In some embodiments, see Fig. 9 , Fig. 9 It is a schematic diagram of the principle of the image generation method provided in the embodiment of the present application. Fig. 9 The model structure shown implements the image generation method provided by the embodiment of the present application. The structure adds a bypass module based on the character action line draft control on the basis of the prediction network. During training, only the newly added control network is trained (the control network includes two multi-head self-attention layers (KQV)), and the original model parameters are not changed, so as to avoid the overfitting problem of retraining the original model under limited data, and also facilitate the migration of the control module of this scheme to other models with the same structure. During character training, the character line draft image is input into the line draft control module, and the representation of a specific character is jointly trained through the text feature module, so that the character representation has a stable generation result on the generative model. Fig. 9 Two sets of data are required for training. One set is the martial arts action image and text data used to train the bypass action line drawing control module, and the other set is the character image and text pairs used to train the specific character generation capability based on line drawing and text features. The newly added control network parameters and character representation in the text representation are trained, and other model structure parameters are not updated.

[0165] In some embodiments, see Fig.10 , Fig.10It is a schematic diagram of the principle of the image generation method provided by the embodiment of the present application, the process of model training control network, for the image-text pair, the image in the image-text pair is converted into a line drawing, and the line drawing is used as input control, and the generation result is close to the original image during training. The text in the image-text pair is converted into text features, and the control network is called to fuse the text features and the line drawing features to obtain the fused features; the prediction network is called to predict the output image j based on the fused features and the text features, and the loss value is obtained based on the output image j and the image in the image pair. Based on the loss value, the initial control network is trained to obtain the control network.

[0166] In some embodiments, see Fig.11 , Fig.11 This is a schematic diagram of the principle of the image generation method provided by the embodiment of the present application, for the overall generation model and training. For the text in the character image, call the feature extraction network to extract features from the text to obtain text features, call the prediction network, and predict the output image j based on the text features; based on the output image j and the image in the character image, obtain the loss value, and based on the loss value, train the feature extraction network to obtain the target feature extraction network.

[0167] In some embodiments, see Figures 9 to 11 , add a character action line control module that can directly control the action to the basic generation model (the prediction network described above), avoid directly fine-tuning the original generation model, which causes the original model to overfit, making it difficult to produce diversified generation results, and helps reduce the potential possibility of overfitting. On the other hand, in order to solve the problem of character consistency, the embodiment of the present application trains a stable character-text feature mapping when embedding character text, so as to achieve the goal of stable character creation. In terms of training methods, the embodiment of the present application adopts sequential training in view of the different functions of the two modules. Since the action module and the character representation model are unrelated, they can be trained separately to reduce the difficulty of training.

[0168] In some embodiments, for line drawing image data, see Fig.12 , Fig.12 is a schematic diagram of the principle of the line drawing image provided by the embodiment of the present application, Fig.12The original image (a puppy on the lawn) and the line drawing image corresponding to the original image (the line drawing of the lawn and the puppy) are shown. For another example, action image-text pairs in martial arts movies and TV series are collected, such as Zhang San doing Tai Chi on the top of the mountain, Li Si doing drunk boxing in the yard, etc. For the images in the image-text pairs, human body detection and face detection are performed to obtain human body / face detection frames. The line drawing is extracted by using an operator to obtain the image, and the facial information in the action image is collected: including the angle of the face deviating from the front and the angle of the face facing the left and right sides. Organize the training set: save the triple sample of "text-image-line drawing", organize it into the action inventory used during reasoning, and clear the background outside the human body detection frame (and the human body detection frame) - the image pixels can be directly set to 0. Since the background is not needed, only the action line drawing is needed, so the background needs to be cleared to avoid adverse effects on future generation results. The "action-line drawing after background removal-text description after background removal-facial information" after the background is cleared is stored in the inventory, and the description words are such as Zhang San doing Tai Chi. The "action-line drawing-original image text description-facial information" of the original image is stored in the library, where the original image text description is such as "Zhang San is practicing Tai Chi on the top of the mountain".

[0169] In some embodiments, see Fig.13 , Fig.13 This is a schematic diagram of the principle of the facial image provided by the embodiment of this application. Since the solution requires a facial embedding model for a specific person, in order to ensure that the corresponding face can be accurately generated when a person's name is input, a person's name-face image-text pair is required as the supervisory information for the character representation. Fig.13 As shown, characters and facial images are collected, and facial image information is collected: including the angle at which the face deviates from the front and the angle at which the face faces left or right.

[0170] In some embodiments, during training, the bypass module and the character module are trained separately. When training the bypass module, a "text-image-line drawing" triplet sample is required, and when training the character module, a character image-text pair is required. When training the character module, a number of samples are randomly selected from the character image-text pair for network learning; when training the bypass module, a number of samples are randomly selected from the triplet for learning.

[0171] In some embodiments, see Fig. 9 ,for Fig. 9The prediction network shown in the figure, for the encoding (embedding, VAE encoding is used in the application to map to the latent feature space) of the image after adding noise, obtains the latent space representation at time T through the diffusion process, and restores the image features (i.e., the original image features without adding noise) through T times of denoising process operations. The restored encoding features are decoded by VAE to obtain the image. For text, the text features are obtained through the text branch and then input into the sampling network. As the input features that need to be denoised in the sampling network, diffusion sampling is used to map the features of the noisy image VAE encoding to the latent space representation at time T. The subsequent denoising process of the image learns to fit the noise representation so that the original image minus the noise representation obtains the real image representation, and the real image is obtained through the decoder D.

[0172] In some embodiments, see Fig. 9 In order to use existing image results to control the generation of some actions that are difficult to depict, a control module with existing action line drawings as input is used to inject action style information into the latent space features of the generation model to achieve action control.

[0173] In some embodiments, see Fig.14 , Fig.14 is a schematic diagram of the principle of the image generation method provided in the embodiment of the present application, such as Fig.14 As shown, the input of this module in the forward calculation is the same as the U-Net input, which is the latent space representation at a certain moment. Among them, QKV7, QKV8, QKV9 are exactly the same as QKV1, QKV2, and QKV3 in structure, and the pre-training parameters are also the same. The difference is that the bypass module parameters will be updated during training in the embodiment of the present application, while the module parameters of the sampling network are not updated. The bypass structure and the structure of the sampling network are fused in the following way: bypass a certain output weighted and add it to the output of the corresponding module of the sampling network, that is, the output of the QKV7 module is not only input into QKV8 for normal network forward calculation, but also generates the input of QKV2 according to the formula out_QKV1+a*out_QKV7=input_QKV2. For example, the output of the last structure of the bypass module - the output of the space transformer, is also weighted with the output of the middle layer of the sampling network as the input of the subsequent residual block. When training the action control module, the bypass module is a parameter that needs to be learned. When training character representation, other modules do not need to be learned, and only the character words in the text feature extraction are fine-tuned.

[0174] In some embodiments, the conventional deep learning training process: a total of N rounds (e.g., 100) of iterations are performed on the entire amount of data, and each round of iteration: the entire amount of image data is trained once in the model is called a round of iteration. In each round of iteration, due to the limited video memory resources of the training machine, all samples cannot be input into the model for training at one time, so all samples need to be trained in batches, and each batch of samples is input into the model for forward calculation, backward calculation, and model parameter update.

[0175] In some embodiments, the full amount of samples (M) are composed of bs samples per batch, and a total of M / bs batches are required to complete one round of full sample iteration; a total of K rounds of full sample iteration are trained (variable, such as 10). In a certain batch, firstly, bs triplet samples (original image, text, line drawing image) are randomly extracted from the full amount of image and text samples, where the text is used to generate constraints, the image is used to add to the noise image as the model input x, the image is also used for loss calculation, and the line drawing image is used to control the input of the network. During training: the first batch of the first round is pre-training parameter initialization: the open source trained model (stable-diffusion v1-5) parameters are used for VAE, text_encoder, and U-Net and are not updated in this training, and only the bypass module is trained. The learning rate of 0.0004 is used for initialization, and the learning rate becomes 0.1 times the original after every 5 rounds of learning, and 4 rounds of training are trained. Take a random seed i to generate a noise image and add it to the image in the triplet to get x. Image x is represented in the latent space to generate ZT. The line drawing is input into the control module. The text information is clipped to get the text representation, which is input into the conditioning text. ZT is forward calculated T times under the KV constraint by denoising U-Net to get the Z0 prediction value. The predicted image is obtained after decoding. The loss is calculated: the loss of the predicted image and the original image is calculated, and the total loss of the batch of samples is counted. The SGD random gradient descent algorithm is used to reverse the loss back to the model to get the gradient of the bypass module and update the parameters. All N / bs batches of training are completed and an iteration ends.

[0176] In some embodiments, similar to the main process in the previous step, the learning process of the character embedding generation model is as follows: for the character, this solution generates a new embedding representation for the character (compared to the original text representation generated based on the clip): Fig.11 As shown in the figure, the character representation is input into the sampling network as a constraint, and the new character representation is input into the cross attention network of the generative model as a constraint. During training, other parameters except the character text representation are fixed and do not participate in the training. The forward calculation process is: the noise generated by the model for the random seed i Figure X ,noise Figure XThe latent space vector (ZT) is projected into the latent space through the diffusion process without any information, such as a snowflake image. The latent space vector is subjected to T times of denoising prediction by the sampling network with the new character representation constraint cross attention network, and finally generates the latent space prediction vector Z, which is restored to noise by the decoder D. Figure X The prediction result. At this time, the text representation module of the clip is the structural parameter that needs to be fine-tuned - note that the forward process refers to the application process, and the noise in the training process Figure X The noise graph generated by random seed i + the supervision graph in the image-text pair are used as the input of the diffusion process, the sampling network, and the bypass module do not need to be updated.

[0177] In some embodiments, in the original generation model, the representation of the text is obtained by extracting the text branch of an open source cross-modal graphic representation model. Except for replacing the character representation with a new representation, the main text representation extraction process is unchanged.

[0178] In some embodiments, the MSE loss is calculated by outputting the predicted image and the mean square error of the image of the image-text pair. The expression of the MSE loss can be:

[0179]

[0180] Among them, MSE is used to indicate the MSE loss, y i Used to indicate the pixel value of each point in the image of the image. Used to indicate the predicted pixel value of each point in the image of the image-text pair.

[0181] In some embodiments, see Fig.15 , Fig.15 is a flow chart of the image generation method provided in the embodiment of the present application. The image generation method provided in the embodiment of the present application can be Fig.15 The illustrated steps 201 to 209 are implemented. The image generation method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will be described using the server alone as an example.

[0182] In step 201, the server inputs a facial image.

[0183] In step 202, the server performs facial image training.

[0184] In step 203, the server obtains a facial line drawing library.

[0185] In some embodiments, the facial line drawing library refers to the mapping relationship between the multiple candidate action postures described above and the corresponding candidate reference images, and the candidate reference images include reference objects corresponding to the candidate action postures.

[0186] In step 204, the server inputs the action statement to be generated.

[0187] In some embodiments, the action sentence to be generated, that is, the expected text described above, includes text for describing the action posture of the target object.

[0188] In step 205, the server acts upon the query return from the inventory.

[0189] In some embodiments, the above action query return from the inventory refers to querying the target inventory content including the action posture of the target object from the inventory. The action inventory is queried to see if there is a corresponding action. If there is no corresponding action, conventional text map generation is performed.

[0190] In step 206, the server extracts the closest angle facial line drawing from the character inventory.

[0191] In step 207, the server performs facial processing and replacement.

[0192] In some embodiments, a line drawing is extracted for the returned action image, the closest facial line drawing is found, and the facial parts of the line drawing are replaced with the line drawing that is closest to the user's face after processing.

[0193] In step 208, the server performs character feature replacement.

[0194] In some embodiments, the above step 208 can be implemented in the following ways: text description input and character feature replacement.

[0195] In step 209, the server generates a model and outputs an image.

[0196] In some embodiments, the above step 209 can be implemented in the following manner: based on the action line drawing control input and the random seed, a generative model is generated to output an image.

[0197] In some embodiments, see Fig.15After the model is trained, the user can input a variety of description sentences for the trained characters to generate corresponding images, such as Zhang San begging on the street, Zhang San kneeling on the ground to worship his master, Zhang San doing a horse pose, Zhang San practicing a volley kick, etc. When the system receives a description sentence from the user, it first queries the action inventory to see if there is a corresponding action. If there is no corresponding action, it generates a conventional text image; if there is a corresponding action, it returns the action image. For the returned action image, extract the line drawing, find the closest facial line drawing, and replace the facial parts of the line drawing with the line drawing that is closest to the user's face after processing; input the replaced action line drawing into the control action module, input the description text into the text encoder, use the model of this solution to generate, and return the generated image.

[0198] In some embodiments, after training, for each character's training image, a line drawing is extracted, and the rotation angle of the face is recorded, such as the angle of deviation from the front, the angle of rotation upward or downward, etc. Each character and all of its facial images and facial line drawings are stored in a representation library to obtain a character representation inventory: character name-facial image-facial line drawing-facial information. Each character's name has a specific representation, and all character representations are saved in the inventory: the character sample is characterized by features to generate a character representation, and the average of these representations is taken to obtain the final representation. Each character and the final representation are stored in the representation library to obtain a character representation inventory, which is recorded as: person name-character image-corresponding features-corresponding image information.

[0199] In some embodiments, see Fig.16 , Fig.16 Schematic diagram of the principle of the image generation method provided by the embodiment of the present application. Fig.16 The above figure shows Wang Xiaoming doing drunken boxing. First, map it to the feature space, that is, find n corresponding features for each word (since the words and features here are not one-to-one corresponding, a certain word may have features of different lengths, such as Qianfan may correspond to 5 features), and then generate text representations from the features. At this time, the representation of the character is replaced with a new representation from the inventory, and the representations of other texts remain unchanged, so a new text representation can be obtained. At this time, the character in the representation has been trained and bound to a specific character image and text pair. The feature list is a vocabulary list in the training stage (a total of 49,152 words, and the concepts of words are different. For example, happiness can be happiness and joy. When expressing joy, it can share a word with enjoying, and when expressing happiness, it can share a word with pleasure). The text sentence is mapped to a feature list (note that one word does not correspond to one representation, not one-to-one correspondence), but the feature and the representation are one-to-one corresponding, one feature corresponds to one representation.

[0200] In some embodiments, the head positions of key characters in the pre-collected action line drawings are recorded (head detection or manual annotation of color images is performed, and annotation boxes are recorded) and saved in the line drawing inventory. The inventory records are "action text description-line drawing-head position-head angle-rotation angle." At the same time, the target character inventory is stored for the head part line drawings of all training images of each user, recorded as "character-line drawing-head position-head angle-rotation angle." Among them, the head angle is the angle of the face deviating from the front (orientation), negative to the left and positive to the right. For example, if the side face of the character in the picture is completely facing left, it is -90 degrees. This information can be pre-annotated, or a classification model can be trained to divide the face orientation into 0 forward, -45 left front, -90 left, 90 right, and 45 right front. For the face, the rotation angle of the face along the same plane is also recorded. For example, the right ear is close to the right shoulder and nods for a positive 45 degrees, a smaller angle towards the right shoulder is 15 degrees, and the left ear is close to the left shoulder for a -45 degree angle. For example, see Fig.17 , Fig.17 is a schematic diagram of the effect of the rotation angle provided in the embodiment of the present application, Fig.17 The character shown may be rotated by -15 degrees.

[0201] In some embodiments, see Fig.18 , Fig.18 This is a schematic diagram of the principle of face replacement provided in an embodiment of the present application, and the position of the action line draft face is obtained (output according to the original image size). According to the orientation angle of the line draft face, the target face line draft image with the closest angle and the face position are found from the target face library (output according to the original image size). According to the plane rotation angle of the action line draft face, the target face is rotated to the corresponding angle position (as shown in the plane rotation angle in the figure below - the direction from the chin to the nose tip of the face should be the same as the vertical orange normal angle). The new face position is detected, and the ratio of the user face to the template face is calculated: the ratio of the height of the key points of the template face to the height of the user face h_target / h_user=ratio is calculated; as an alternative, the size of the action training face and the user face frame can also be used as the ratio (but it is not accurate enough). Subtract the overall user face image size * ratio to obtain the result image of the new user face angle and size alignment. Clear the face frame of the action line draft image, and move the user face line draft image in (move the chin position to the chin position of the template face). As an alternative, for the result image after size alignment, using an action template image to stylize it can make the generated result more natural, but since the line drawings are not much different, stylization is not necessary.

[0202] In some embodiments, see Fig.18, find the stock face with the closest face angle, obtain the rotated face by plane rotating the stock face, detect the new face position in the rotated face, scale the new face position, and perform stylization processing, replace the image style of the action line drawing, and obtain the updated face.

[0203] It is understandable that in the embodiments of the present application, related data such as expected text is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0204] The following is a description of an exemplary structure of the image generation device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the image generating device 455 of the memory 450 may include: a text acquisition module 4551, used to acquire an expected text for describing the expectation corresponding to the generated target image, wherein the expected text includes text for describing the action posture of the target object; an image acquisition module 4552, used to acquire a reference image of the expected text, wherein the action posture of the reference object in the reference image matches the action posture described in the text; a replacement module 4553, used to replace the face of the reference object in the reference image with the face of the target object when the reference object is different from the target object, so as to obtain a target reference image; and a prediction module 4554, used to combine the expected text and the target reference image to predict a target image that meets the expectation.

[0205] In some embodiments, the above-mentioned image acquisition module 4552 is used to obtain the mapping relationship between multiple candidate action postures and corresponding candidate reference images, where the candidate reference images include reference objects of the corresponding candidate action postures; the action posture of the target object is matched with each candidate action posture respectively to obtain the matching degree between the action posture of the target object and each candidate action posture, and the candidate action posture corresponding to the highest matching degree is determined as the target action posture that matches the action posture of the target object; based on the mapping relationship, the candidate reference image corresponding to the target action posture is determined as the reference image.

[0206] In some embodiments, the above-mentioned replacement module 4553 is used to obtain the facial image of the target object and the image size of the facial image; determine the facial angle of the reference object in the reference image and the facial size of the reference object in the reference image; adjust the image size of the facial image to the facial size to obtain a resized image of the facial image; obtain the facial angle of the target object in the resized image, and adjust the facial angle of the target object in the resized image to the facial angle of the reference object to obtain a target map; replace the image content of the area where the face of the reference object in the reference image is located with the target map to obtain a target reference image.

[0207] In some embodiments, the above-mentioned prediction module 4554 is also used to obtain target text features of the expected text, the target text features are used to indicate at least one historical action posture corresponding to the target object; perform image feature extraction on the target reference image to obtain image features of the target reference image; perform feature fusion on the image features and the target text features at least once to obtain fusion features corresponding to each feature fusion; combine the target text features and each fusion feature to predict a target image that meets expectations.

[0208] In some embodiments, the above-mentioned prediction module 4554 is also used to extract text features of the expected text to obtain expected text features of the expected text, the expected text features including identification text features for identifying the target object; obtain updated text features of the target object, the updated text features are used to indicate at least one historical action posture corresponding to the target object; replace the identification text features in the expected text features with the updated text features to obtain the target text features.

[0209] In some embodiments, the above-mentioned prediction module 4554 is also used to obtain the mapping relationship between multiple candidate objects and corresponding text features, and the multiple candidate objects include a target object; query the target mapping relationship including the target object from the mapping relationship, and determine the text features in the target mapping relationship as updated text features.

[0210] In some embodiments, the above-mentioned prediction module 4554 is also used to perform feature fusion on image features and target text features to obtain a first fused feature when the number of feature fusions is multiple times; perform feature fusion on the first fused feature and the target text feature at least once to obtain fused features corresponding to each feature fusion.

[0211] In some embodiments, the prediction module 4554 is further used to traverse i and perform the following processing: perform feature fusion on the i-th fusion feature and the target text feature to obtain the i+1-th fusion feature; wherein 1≤i≤N-1, and N is the maximum number of feature fusions.

[0212] In some embodiments, the above-mentioned prediction module 4554 is also used to obtain an image prediction network, which includes a downsampling network, an upsampling network and a prediction network; calling the downsampling network to downsample the target text features to obtain reference features, and fusing the reference features with the fused features obtained by the last feature fusion to obtain target fused features; calling the upsampling network to upsample the target fused features to obtain target features; calling the prediction network to predict a target image that meets expectations based on the target features.

[0213] In some embodiments, the downsampling network includes at least one downsampling layer, and the number of downsampling layers is the same as the number of feature fusions. The above-mentioned prediction module 4554 is also used to call the first downsampling layer when the number of feature fusions is multiple times, downsample the target text feature, and obtain the first reference feature; traverse j to perform the following processing: fuse the jth reference feature and the fusion feature corresponding to the jth feature fusion to obtain the jth target fusion feature; call the jth downsampling layer, downsample the jth target fusion feature, and obtain the j+1th reference feature; wherein 1≤j≤M-1, M is the number of downsampling layers in the downsampling network; determine the Mth reference feature as the reference feature.

[0214] In some embodiments, the above-mentioned prediction module 4554 is also used to obtain an initial image prediction network, a target image sample, and an expected text sample for describing the expectation corresponding to the generated target image sample; calling the initial image prediction network, based on the expected text sample, predicting a first predicted image, and determining a first loss value based on the first predicted image and the target image sample; obtaining a reference image sample of the expected text sample, and calling the initial image prediction network, combining the reference image sample and the expected text sample, predicting a second predicted image, and determining a second loss value based on the second predicted image and the target image sample; training the initial image prediction network based on the first loss value and the second loss value to obtain an image prediction network.

[0215] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction or the computer program from the computer-readable storage medium, and the processor executes the computer executable instruction or the computer program, so that the electronic device performs the image generation method described in the embodiment of the present application.

[0216] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions or computer programs, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the image generation method provided by the present application embodiment, for example, Figure 3 The image generation method is shown.

[0217] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEP ROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various electronic devices including one or any combination of the above memories.

[0218] In some embodiments, computer executable instructions or computer programs may be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines or other units suitable for use in a computing environment.

[0219] As an example, computer-executable instructions or a computer program may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0220] As an example, computer executable instructions or a computer program may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0221] In summary, the embodiments of the present application have the following beneficial effects:

[0222] (1) By obtaining an expected text for describing the expectation corresponding to the generated target image and a reference image of the expected text, when the reference object in the reference image is different from the target object in the expected text, the face of the reference object in the reference image is replaced with the face of the target object to obtain a target reference image, and the target text and the target reference image are combined to predict a target image that meets the expectation. In this way, by obtaining a reference image that matches the action posture described in the text, and replacing the face of the reference object in the reference image with the face of the target object, the face of the object in the obtained target reference image is the face of the target object, and the action posture also matches the action posture described in the text, so that the target reference image can basically meet the action posture of the target object described in the expected text, and thus the expected text and the target reference image are combined to predict that the target image that meets the expectation can better fit the action posture of the target object described in the expected text, thereby effectively improving the information fit between the expected text and the target image.

[0223] (2) Since the action posture of the reference object in the reference image matches the action posture described in the text, the reference image provides strong image data support for the expected text. The reference image can provide a matching image description for the expected text. By combining the expected text and the reference image, a target image that meets the expectations is predicted. The obtained predicted image can be more consistent with the action posture described in the text in terms of action posture, which effectively improves the information fit between the expected text and the target image.

[0224] (3) By matching the action posture of the target object with each candidate action posture respectively, the candidate action posture corresponding to the highest matching degree is determined as the target action posture that matches the action posture of the target object. Since the candidate reference image includes the reference object of the corresponding candidate action posture, by determining the candidate reference image corresponding to the target action posture as the reference image, the action posture of the corresponding reference object included in the reference image has a high degree of similarity with the action posture of the target object. The reference image provides strong image data support for the expected text, and can provide a suitable image description for the expected text through the reference image. By combining the expected text and the reference image, the target image that meets the expectation is predicted, and the obtained predicted image can be more consistent with the action posture described in the text in terms of action posture, which effectively improves the information fit between the expected text and the target image.

[0225] (4) A target reference image is obtained by replacing the face of a reference object in a reference image with the face of a target object, so that the action posture of the corresponding reference object included in the target reference image has a high degree of similarity with the action posture of the target object, and the facial features of the corresponding reference image included in the target reference image are completely identical to the facial features of the target object. The target reference image can provide a matching image description for the expected text, and a target image that meets the expectations is predicted by combining the expected text and the target reference image. The obtained predicted image can be more consistent with the action posture and facial features described in the text in terms of action posture, thereby effectively improving the information fit between the expected text and the target image.

[0226] (5) Since the line graph corresponding to the target reference image can effectively reduce the redundant information in the target reference image and retain the valid information in the target reference image, therefore, line extraction is performed on the target reference image to obtain the line graph corresponding to the target reference image, and feature extraction is performed on the line graph to obtain the image features of the target reference image, which can effectively reduce the information redundancy of the image features.

[0227] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire an expected text for describing the expectation corresponding to generating the target image, wherein the expected text includes text for describing the action posture of the target object; Acquire a reference image of the expected text, wherein the motion posture of a reference object in the reference image matches the motion posture described in the text; When the reference object is different from the target object, replacing the face of the reference object in the reference image with the face of the target object to obtain a target reference image; The expected text and the target reference image are combined to predict a target image that meets the expectation.

2. The method according to claim 1, characterized in that: The step of obtaining a reference image of the expected text includes: Acquire mapping relationships between a plurality of candidate action postures and corresponding candidate reference images, wherein the candidate reference images include reference objects corresponding to the candidate action postures; Matching the target object's motion posture with each of the candidate motion postures respectively to obtain a matching degree between the target object's motion posture and each of the candidate motion postures, and determining the candidate motion posture corresponding to the highest matching degree as the target motion posture matching the target object's motion posture; Based on the mapping relationship, the candidate reference image corresponding to the target action posture is determined as the reference image.

3. The method according to claim 1, characterized in that The step of replacing the face of the reference object in the reference image with the face of the target object to obtain a target reference image comprises: Acquire a facial image of the target object and an image size of the facial image; Determining a facial angle of the reference object in the reference image, and a facial size of the reference object in the reference image; resizing the image size of the facial image to the facial size to obtain a resized image of the facial image; Acquire the facial angle of the target object in the resized image, and adjust the facial angle of the target object in the resized image to the facial angle of the reference object to obtain a target map; The image content of the area where the face of the reference object in the reference image is located is replaced with the target map to obtain the target reference image.

4. The method according to claim 1, characterized in that: The combining the expected text and the target reference image to predict a target image that meets the expectation includes: Acquire a target text feature of the expected text, wherein the target text feature is used to indicate at least one historical action gesture corresponding to the target object; Performing image feature extraction on the target reference image to obtain image features of the target reference image; Performing feature fusion on the image feature and the target text feature at least once to obtain fusion features corresponding to each feature fusion; The target text feature and each of the fusion features are combined to predict a target image that meets the expectation.

5. The method according to claim 4, characterized in that The step of obtaining target text features of the expected text includes: Performing text feature extraction on the expected text to obtain expected text features of the expected text, wherein the expected text features include identification text features for identifying the target object; Acquire an updated text feature of the target object, where the updated text feature is used to indicate at least one historical action gesture corresponding to the target object; The identification text feature in the expected text feature is replaced by the update text feature to obtain the target text feature.

6. The method according to claim 5, characterized in that The obtaining of the updated text feature of the target object includes: Acquire mapping relationships between a plurality of candidate objects and corresponding text features, respectively, wherein the plurality of candidate objects include the target object; A target mapping relationship including the target object is queried from the mapping relationship, and the text feature in the target mapping relationship is determined as the update text feature.

7. The method according to claim 4, characterized in that The performing at least one feature fusion on the image feature and the target text feature to obtain fusion features corresponding to each feature fusion, includes: When the number of times of feature fusion is multiple times, the image feature and the target text feature are fused to obtain a first fused feature; The first fusion feature and the target text feature are fused at least once to obtain fusion features corresponding to each feature fusion.

8. The method according to claim 7, characterized in that The performing at least one feature fusion on the first fusion feature and the target text feature to obtain fusion features corresponding to each feature fusion, includes: Traversing i, the following processing is performed: feature fusion is performed on the i-th fusion feature and the target text feature to obtain the i+1-th fusion feature; Wherein, 1≤i≤N-1, N is the maximum number of feature fusions.

9. The method according to claim 4, characterized in that The step of combining the target text feature and each of the fusion features to predict a target image that meets the expectation includes: Acquire an image prediction network, wherein the image prediction network includes a downsampling network, an upsampling network, and a prediction network; Calling the downsampling network to downsample the target text feature to obtain a reference feature, and fusing the reference feature with a fused feature obtained by the last feature fusion to obtain a target fused feature; Calling the upsampling network to upsample the target fusion feature to obtain the target feature; The prediction network is called to predict a target image that meets the expectation based on the target features.

10. The method according to claim 9, characterized in that The downsampling network includes at least one downsampling layer, the number of the downsampling layers is the same as the number of feature fusions, and the calling of the downsampling network to downsample the target text feature to obtain a reference feature includes: When the number of times of feature fusion is multiple times, calling the first downsampling layer to downsample the target text feature to obtain the first reference feature; Traversing j, the following processing is performed: fusing the jth reference feature with the fusion feature corresponding to the jth feature fusion to obtain the jth target fusion feature; Call the j-th downsampling layer to downsample the j-th target fusion feature to obtain the j+1-th reference feature; Wherein, 1≤j≤M-1, M is the number of the downsampling layers in the downsampling network; The Mth reference feature is determined as the reference feature.

11. The method according to claim 9, characterized in that The obtaining of the image prediction network comprises: Obtaining an initial image prediction network, a target image sample, and an expected text sample for describing the expected value corresponding to generating the target image sample; Calling the initial image prediction network, predicting a first predicted image based on the expected text sample, and determining a first loss value based on the first predicted image and the target image sample; Obtaining a reference image sample of the expected text sample, calling the initial image prediction network, combining the reference image sample and the expected text sample, predicting a second predicted image, and determining a second loss value based on the second predicted image and the target image sample; Based on the first loss value and the second loss value, the initial image prediction network is trained to obtain the image prediction network.

12. An image generating device, characterized in that: The device comprises: A text acquisition module, used to acquire an expected text for describing the expected corresponding to the generated target image, wherein the expected text includes a text for describing the action posture of the target object; An image acquisition module, used to acquire a reference image of the expected text, wherein the action posture of the reference object in the reference image matches the action posture described in the text; a replacement module, configured to replace the face of the reference object in the reference image with the face of the target object to obtain a target reference image when the reference object is different from the target object; The prediction module is used to combine the expected text and the target reference image to predict a target image that meets the expectation.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the image generation method according to any one of claims 1 to 11 when executing the computer executable instructions or computer program stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the image generating method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the image generating method according to any one of claims 1 to 11 is implemented.

Images