Virtual image generation method and related device, electronic device, and storage medium
By acquiring user description text and image features, and using pre-trained language models and image generation models for iterative optimization, the time and cost issues in generating personalized virtual images have been solved, achieving efficient and low-cost generation of personalized virtual images.
Patent Information
- Application Number
- CN202210395741.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-14
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-04-14
AI Technical Summary
Existing technologies for generating personalized virtual avatars suffer from problems such as long production time, high cost, and high hardware requirements.
By acquiring user description text and several image features, candidate virtual images are generated, and users are allowed to select a target virtual image that meets their personalized needs. The pre-trained language model and image generation model are used for iterative optimization to finally generate a virtual image that meets the user's expectations.
It reduces the time and cost of generating virtual avatars, lowers the requirements for hardware equipment, and improves generation efficiency and accuracy.
Smart Images

Figure CN114913303B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method for generating a virtual image and related devices, electronic equipment, and storage media. Background Art
[0002] In recent years, mixed reality products represented by HoloLens, Quest2, Google Glass, etc. have gradually entered the public eye. As an important carrier of information expression in the virtual world, virtual images have blossomed in online scenarios such as smartphone assistants, virtual e-commerce anchors, virtual celebrities, and offline scenarios such as banks, subways, and hospitals, and are showing an expansion trend.
[0003] At present, avatar generation mainly uses CG (Computer Graphics) technology modeling. If you want to achieve personalization, you need to modify it from the modeling, which takes a long time to make, has high labor costs, and has high requirements for equipment hardware. In view of this, how to reduce the time and cost of avatar generation and reduce the requirements for hardware equipment while meeting the needs of personalized avatar generation has become an urgent problem to be solved. Summary of the invention
[0004] The main technical problem solved by this application is to provide a virtual image generation method and related devices, electronic devices, and storage media, which can improve the accuracy of answering questions.
[0005] In order to solve the above technical problems, the first aspect of the present application provides a virtual image generation method, including: obtaining a user's description text of a desired virtual image, and obtaining a number of image features; generating an image based on the distribution of each image feature and the text features of the description text to obtain a candidate virtual image; in response to the user selecting a candidate virtual image as a target virtual image, obtaining a feature representation that conforms to the feature distribution of the target virtual image as a new image feature; and generating a final virtual image based on the new image features and text features.
[0006] In order to solve the above-mentioned technical problems, the second aspect of the present application provides a virtual image generation device, including: an acquisition module, a generation module, a response module and a loop module, the acquisition module is used to obtain the user's description text of the desired virtual image and obtain a number of image features; the generation module is used to generate an image based on each image feature and the text features of the description text, so as to obtain a candidate virtual image; the response module is used to respond to the user selecting a candidate virtual image as a target virtual image, and obtain a feature representation that conforms to the feature distribution of the target virtual image as a new image feature; the loop module is used to generate a final virtual image based on the new image features and text features.
[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions to implement the virtual image generation method of the first aspect.
[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the virtual image generation method of the first aspect.
[0009] The above scheme obtains the user's description text of the desired virtual image and obtains several image features, and then generates an image based on each image feature and the text features of the description text to obtain a candidate virtual image, and in response to the user selecting the candidate virtual image as the target virtual image, obtains a feature representation that conforms to the feature distribution of the target virtual image as a new image feature, and generates the final virtual image based on the new image feature and text features. That is to say, in the process of generating a virtual image, on the one hand, it is not necessary to rely on computer graphics modeling, and on the other hand, because several image features are obtained in the initial stage to generate candidate virtual images that conform to various feature distributions, the image is generated again in the subsequent stage through the candidate virtual image selected by the user, so as to meet the user's personalized expectations as much as possible. Therefore, on the basis of meeting the personalized virtual image generation needs, it is possible to reduce the time and cost of virtual image generation and reduce the requirements for hardware equipment. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 It is a flowchart of an embodiment of a method for generating a virtual image of the present application;
[0011] Figure 2 It is a schematic diagram of the framework of an embodiment of an image generation model;
[0012] Figure 3 is a flow chart of an embodiment of a training image generation model;
[0013] Figure 4 is a schematic diagram of a process of training an embodiment of an image generation model;
[0014] Figure 5 It is a schematic diagram of the framework of an embodiment of a virtual image generating device of the present application;
[0015] Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;
[0016] Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION
[0017] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0018] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0019] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.
[0020] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a method for generating a virtual image of the present application. Specifically, the method may include the following steps:
[0021] Step S11: Obtain the user's description text of the desired virtual image and obtain several image features.
[0022] In an implementation scenario, it should be noted that, unless otherwise specified, the "virtual image" described in the embodiments of the present disclosure (such as the expected virtual image, the final virtual image, the sample virtual image, etc.) may include but is not limited to: people, animals, plants, etc., and are not limited here. For example, when applied to scenarios such as early childhood education, the "virtual image" may be a cute child; when applied to scenarios such as animal protection, the "virtual image" may be a cute little squirrel; when applied to scenarios such as plant protection, the "virtual image" may be a tall sapling. Other situations can be deduced by analogy, and examples are not given one by one here.
[0023] In an implementation scenario, it should be noted that the "virtual image" described in the embodiments of the present disclosure (such as the expected virtual image, the final virtual image, the sample virtual image, etc.) can be represented by image data (such as two-dimensional image data, three-dimensional image data, etc.). Specifically, the lines and colors of the virtual image can be expressed by RGB image data. In addition, the "virtual image" described in the embodiments of the present disclosure can also be represented by mesh data (such as triangular mesh data). Specifically, the three-dimensional outline of the virtual image can be expressed by triangular mesh data. The specific expression method of the specific "virtual image" can be set according to the actual situation and is not limited here.
[0024] In one implementation scenario, before formally generating a virtual image, the user can describe the desired virtual image through text to define the various attributes of the desired virtual image (such as whether it is a person or an object, what style, etc.). For example, in the case where the desired virtual image is a "cute child", the description text can be "cute child", or the description text can be more specific as "cute little girl" or "cute little boy", etc.; or, in the case where the desired virtual image is a "cute little squirrel", the description text can be "cute little squirrel", and in the case where the desired virtual image is a "tall sapling", the description text can be "tall sapling". Other situations can be deduced by analogy, and examples are not given one by one here. It can be understood that in real scenarios, the more accurate and specific the description text of the desired virtual image is, the more helpful it is to quickly generate a final virtual image that is as close to the desired virtual image as possible.
[0025] In one implementation scenario, in order to obtain as diverse candidate virtual images as possible after the first image generation, so that the candidate virtual images include those that are as close to the desired virtual image as possible to meet the needs of subsequent user selection, the image features may respectively conform to the feature distribution of various types of virtual images. In addition, the specific number of the image features may also be set as much as possible, such as 2, 3, 4, 5, etc., which are not limited here.
[0026] Exemplarily, as a possible implementation, different types of sample virtual images may be collected first, and then features of each sample virtual image may be extracted respectively to obtain the above-mentioned image features.
[0027] Exemplarily, as another possible implementation, several candidate feature distributions may be acquired in advance, and the several candidate feature distributions respectively characterize the feature distributions of different types of virtual images. On this basis, the above-mentioned several image features may be obtained by sampling from each candidate feature distribution respectively. It should be noted that, when the "virtual image" is a person, several candidate feature distributions may include feature distributions of virtual images of different races, clothing, hair accessories, backgrounds, etc.; or, when the "virtual image" is an animal, several candidate feature distributions may include feature distributions of virtual images of different species, fur colors, backgrounds, etc.; or, when the "virtual image" is a plant, several candidate feature distributions may include feature distributions of virtual images of different species, leaf colors, flower colors, etc. Other situations can be deduced by analogy, and examples are not given one by one here.
[0028] In one implementation scenario, the feature distribution may include but is not limited to the feature mean, feature variance, etc., which are not limited here. For example, in the case where the image features of the virtual image are subsequently represented by a feature vector with a dimension of N, the feature distribution may include but is not limited to the feature mean and feature variance of each dimension of the feature, which are not limited here.
[0029] In one implementation scenario, as mentioned above, the candidate feature distribution may include, but is not limited to, parameters such as the feature mean and feature variance of each dimensional feature. On this basis, in the process of sampling from each candidate feature distribution to obtain an image feature, the feature element of that dimension may be sampled based on the feature mean and feature variance of each dimensional feature of the candidate feature distribution, and the feature elements of each dimensional feature sampled from the candidate feature distribution may be combined as the image feature sampled from the candidate feature distribution. Still taking the example that the candidate feature distribution contains the feature mean and feature variance of D dimensional features, when sampling the feature element of the i-th dimension, Random sampling of the interval obtains the characteristic elements of the i-th dimension, where represents the feature mean of the i-th dimension feature in the j-th candidate feature distribution, It represents the feature variance of the i-th dimension feature in the j-th candidate feature distribution. On this basis, for each candidate feature distribution, a D-dimensional image feature can be obtained from its sampling.
[0030] In an implementation scenario, several sample virtual images can be collected in advance, and the distribution of each candidate feature can be obtained based on the sample virtual images. Specifically, the sample image features of each sample virtual image can be extracted based on principal component analysis, and then the central features of the preset value cluster centers can be obtained based on the pre-set value principal components obtained by performing principal component analysis. On this basis, the sample image features can be clustered based on each central feature to obtain the candidate feature distributions that characterize different types of virtual images. It should be noted that the mathematical definition of principal component analysis (PCA) is an orthogonal linear transformation that transforms the data into a new coordinate system so that the first largest variance of any projection of this data is on the first coordinate (called the first principal component), the second largest variance is on the second coordinate (the second principal component), and so on. The algorithm flow of principal component analysis can refer to the technical details of principal component analysis, which will not be repeated here. The above method first extracts the sample virtual image based on principal component analysis, and then clusters it through a preset number of principal components to obtain the candidate feature distribution that characterizes virtual images of different virtual types. Therefore, feature clustering can be performed in an unsupervised manner, which helps to improve the accuracy of the candidate feature distribution.
[0031] In a specific implementation scenario, in order to further improve the accuracy of the candidate feature distribution, the sample virtual image may be preprocessed before the principal component analysis is used for feature extraction. Taking the sample virtual image represented by image data as an example, the image data may be subjected to feature point alignment processing so that the sample virtual image is uniformly located at a preset position (such as the middle position) of the image data. For example, when the sample virtual image is a person, the facial data may be subjected to feature point alignment so that the positions of the facial features in the image are consistent (such as, all in the middle position). For ease of description, the aligned image may be denoted as I align m , where m ranges from 0 to M, and M is the total number of sample virtual images. On this basis, the sample image features can be extracted based on principal component analysis.
[0032] In a specific implementation scenario, each sample virtual image can be encoded to obtain sample encoding features, and then the sample encoding features can be projected into an orthogonal space based on principal component analysis to obtain sample image features. Still taking the sample virtual image as a person represented by image data as an example, a pre-trained neural network (e.g., a face recognition network) can be used to encode the sample virtual image (e.g., the aligned image I align m ) is processed, and the output of the last network layer (such as the fully connected layer) of the neural network is taken as the sample encoding feature F m , this feature can be expressed as a 1*1*D dimensional vector, where D can be set according to the actual situation, such as 128, 256, etc., which is not limited here. On this basis, in order to make each cluster center more distinguishable, the sample encoding features can be projected into a mutually orthogonal hyperdimensional space through principal component analysis, so as to ensure that the total energy of the projected vector is not less than 99%. For the convenience of description, the sample image feature can be expressed as F pca m In the above method, each sample virtual image is encoded to obtain the sample encoding features, and the sample encoding features are reprojected to the orthogonal space based on principal component analysis to obtain the sample image features, which can make each cluster center more distinguishable, thereby improving the accuracy of the candidate feature distribution.
[0033] In a specific implementation scenario, after obtaining the sample image features, a Gaussian Mixture Model (GMM) can be used for unsupervised clustering. In addition, in order to make the virtual image generated based on the candidate feature distribution more abundant, the central features of the cluster center can be determined according to the preset value of the principal component. Specifically, the preset value C can be set according to the actual situation, such as 10, 20, etc., which is not limited here. On this basis, the eigenvalue of each principal component can be multiplied by the eigenvector as the central feature of one of the cluster centers. Based on the central features of the C cluster centers, the Gaussian Mixture Model can be used for clustering to obtain C candidate feature distributions. As mentioned above, the candidate feature distribution may include but is not limited to parameters such as the feature mean and feature variance of each dimensional feature, and the parameters of each cluster center of the Gaussian Mixture Model can be obtained by the Expectation-Maximization (EM) algorithm. For the specific process of parameter acquisition, please refer to the technical details of the maximum expectation algorithm, which will not be repeated here.
[0034] Step S12: generating an image based on each image feature and the text features of the description text to obtain a candidate virtual image.
[0035] In an implementation scenario, in order to improve the accuracy of text features, the description text can be first segmented and keywords extracted, and the keywords can be combined to obtain a new description text. On this basis, the new description text is feature extracted based on a pre-trained language model such as BERT (Bidirectional Encoder Representation from Transformers) to obtain text features. In addition, the specific extraction process of text features can refer to the technical details of pre-trained language models such as BERT, which will not be repeated here.
[0036] In a specific implementation scenario, for example, taking the description text "a cute little girl is playing on the beach" as an example, it can be first segmented and the keyword "cute little girl" can be extracted to obtain a new description text "cute little girl".
[0037] In a specific implementation scenario, after extraction by the pre-trained language model, a text feature with a dimension of 1*S*D can be obtained, where S represents the code length (i.e., the length of the new description text) and D represents the encoding dimension. For example, taking the new description text containing n keywords as an example, the i-th keyword in the new description text can be recorded as K i , the encoding vector corresponding to the keyword can be recorded as Tc i. Other situations can be deduced in the same way, and no examples are given here one by one.
[0038] It should be noted that, in the case of obtaining image features through sampling of candidate feature distribution, in real scenarios, feature sampling can be performed first and then text features can be extracted, or text features can be extracted first and then feature sampling can be performed, or feature sampling and text feature extraction can be performed simultaneously, and no limitation is made here. In addition, after obtaining a number of image features and text features, image generation can be performed based on the image features and text features to obtain candidate virtual images, and then the above image generation can be performed for each image feature to obtain the corresponding candidate virtual image.
[0039] In an implementation scenario, in order to improve the efficiency of generating virtual images, an image generation model can be pre-trained, so that candidate virtual images can be generated by the image generation model. Specifically, the image generation model can be trained based on several sample virtual images, and the sample virtual images can be annotated with sample description texts. It should be noted that the specific meaning of the sample virtual image can refer to the above-mentioned related descriptions, which will not be repeated here. In addition, the image generation model may include but is not limited to several sequentially connected network layers (such as deconvolution layers), and the network structure of the image generation model is not limited here. The training process of the image generation model can refer to the following public embodiments, which will not be repeated here. In the above manner, the candidate virtual image is generated by the image generation model, the image generation model is trained based on several sample virtual images, and the sample virtual image is annotated with sample description texts. On the one hand, the efficiency of generating virtual images can be improved. On the other hand, since the candidate feature distribution is also obtained based on several sample virtual images, the data consistency of the two processes of model training and feature clustering can be maintained, which is conducive to improving the accuracy of generating virtual images.
[0040] In a specific implementation scenario, for each image feature obtained by sampling the candidate feature distribution, it can be spliced with the text feature describing the text to obtain a spliced feature. Exemplarily, still taking the example of a feature vector in which the text feature is represented as a 1*S*D dimension and the image feature is represented as a 1*1*D dimension, a spliced feature of 1*1*(S+1)*D dimension can be obtained after splicing the two. On this basis, the spliced feature can be feature mapped to obtain a preset number of mapping features. It should be noted that the image generation network can include a preset number of network layers (such as a deconvolution layer), and each mapping feature can be respectively input into the network layer of the image generation network to control the content synthesis of network layers at different levels, and finally a candidate virtual image that conforms to the candidate feature distribution can be generated.
[0041] In a specific implementation scenario, please refer to Figure 2 , Figure 2 Schematic diagram of the framework of an embodiment of the image generation model. Figure 2 As shown, the image generation model may include a mapping network and a generation network, and the mapping network may include n sequentially connected fully connected (FC) layers and m parallel connected fully connected layers. Based on this, after the concatenated features pass through n sequentially connected fully connected layers, a first mapping feature W can be obtained, and after the first mapping feature W passes through m (e.g., 18) sequentially connected fully connected layers, m second mapping features, i.e., the aforementioned mapping features, can be obtained. In addition, the generation network may include k sequentially connected network layers (e.g., deconvolution layers), and the aforementioned m mapping features are respectively input into different network layers to control the synthesis of image content at different levels, thereby obtaining a candidate virtual image. It should be noted that Figure 2 What is shown is only a possible implementation of the image generation model in actual application, and the specific structure of the image generation model is not limited thereto.
[0042] Step S13: In response to the user selecting a candidate virtual image as the target virtual image, a feature representation that matches the feature distribution of the target virtual image is obtained as a new image feature.
[0043] In one implementation scenario, the user may select at least one candidate virtual image that is close to the desired virtual image as the target virtual image to continue generating the virtual image, so that the virtual image to be generated further is closer to the desired virtual image. Of course, before the subsequent generation, it is necessary to first obtain a feature representation that conforms to the feature distribution of the target virtual image as a new image feature. For example, please refer to Figure 2 , taking a total of 5 candidate feature distributions as an example, for ease of description, they can be recorded as candidate feature distribution No. 1, candidate feature distribution No. 2, candidate feature distribution No. 3, candidate feature distribution No. 4 and candidate feature distribution No. 5, respectively. After the above processing, candidate virtual image No. 1, candidate virtual image No. 2, candidate virtual image No. 3, candidate virtual image No. 4 and candidate virtual image No. 5 can be generated respectively. In one possible case, the user can select one of the candidate virtual images, such as selecting candidate virtual image No. 2, as the target virtual image, and at this time, the feature representation that conforms to the candidate feature distribution No. 2 can be obtained as a new image feature; or, in another possible case, the user can select multiple (such as two, three, four, etc.) candidate virtual images, such as selecting candidate virtual image No. 2 and candidate virtual image No. 3, respectively as the target virtual image, and at this time, the feature representation that conforms to the candidate feature distribution No. 2 can be obtained as a new image feature, and the feature representation that conforms to the candidate feature distribution No. 3 can be obtained as a new image feature. Other situations can be deduced by analogy, and no examples are given here one by one.
[0044] In a specific implementation scenario, in order to obtain a feature representation that conforms to the feature distribution of the target virtual image to obtain a new image feature, a new information feature can be obtained by interpolation based on the target feature. It should be noted that the target feature is the image feature of the target virtual image generated. For example, taking the aforementioned selection of candidate virtual image No. 2 as the target virtual image as an example, the image feature sampled from the feature distribution of candidate No. 2 can be used as the target feature. Other situations can be deduced by analogy, and examples are not given one by one here. In addition, for the specific process of interpolation based on the target feature, reference can be made to the technical details of interpolation algorithms such as linear interpolation and spherical interpolation, which will not be repeated here. In the above method, a new image feature is obtained based on the interpolation of the target feature, and the target feature is the image feature of the target virtual image generated, so that the feature representation that conforms to the feature distribution of the target virtual image can be obtained directly through the interpolation algorithm, thereby improving the convenience of generating new image features.
[0045] In a specific implementation scenario, different from the feature representation of the feature distribution that conforms to the target virtual image obtained by the interpolation algorithm, as mentioned above, several image features can be sampled from several candidate feature distributions, and several candidate feature distributions respectively characterize the feature distributions of different types of virtual images, then new image features can also be sampled from the target feature distribution. It should be noted that the target feature distribution is the candidate feature distribution of the target feature obtained by sampling. Exemplarily, taking the aforementioned selection of candidate virtual image No. 2 as the target virtual image as an example, the candidate feature distribution No. 2 can be used as the target feature distribution. Other situations can be deduced by analogy, and examples are not given one by one here. In addition, the specific process of sampling based on the target feature distribution can refer to the relevant description of sampling based on the candidate feature distribution, which will not be repeated here. In the above manner, new image features are obtained by sampling the target feature distribution, so that the feature representation of the feature distribution that conforms to the target virtual image can be obtained by resampling, thereby improving the accuracy of generating new image features.
[0046] In one implementation scenario, instead of the user selecting at least one candidate virtual image that is close to the desired virtual image as the target virtual image, the user may also directly determine at least one of the candidate virtual images as the final virtual image. That is, in actual application, there may be a virtual image that meets the user's expectations among the candidate virtual images generated in the first round, and the user may directly select it as the final virtual image. In this case, the subsequent process may be terminated directly.
[0047] Step S14: Generate a final virtual image based on the new image features and text features.
[0048] In one implementation scenario, a new candidate virtual image can be generated based on the new image features and text features. For the specific generation process, please refer to the above description of generating a candidate virtual image based on the image features and text features of the description text, which will not be repeated here. On this basis, the new candidate virtual image can be directly used as the final virtual image.
[0049] In one implementation scenario, in order to make the final virtual image infinitely close to the desired virtual image, the aforementioned steps of generating candidate virtual images based on the image features and the text features of the description text and subsequent steps can be re-executed based on the new image features, until the user selects at least one candidate virtual image as the final virtual image. That is, after obtaining the new image features, the aforementioned steps of generating candidate virtual images based on the image features and the text features of the description text can be executed again. At this time, if the user selects the candidate virtual image as the final virtual image, the process can be terminated. On the contrary, if the user selects the candidate virtual image as the new target virtual image, the feature representation that conforms to the feature distribution of the new target virtual image can be obtained as the new image features, and the above process can be executed again in a loop. The above method, based on the new image features, re-executes the steps of generating candidate virtual images based on each image feature and the text features of the descriptive text and the subsequent steps, until the user selects at least one candidate virtual image as the final virtual image. Therefore, by generating virtual images through multiple iterations, the generated virtual image can be infinitely close to the user's expected virtual image. This is different from the method of pre-constructing a database containing a limited number of candidate virtual images and selecting the final virtual image from the database. By re-acquiring new image features during each iteration, a completely new virtual image can be generated during each iteration, thereby maximizing the richness of the candidate virtual images and increasing the possibility of the user selecting a final virtual image that is infinitely close to the expected virtual image, which helps to greatly improve user satisfaction.
[0050] The above scheme obtains the user's description text of the desired virtual image and obtains several image features, and then generates an image based on each image feature and the text features of the description text to obtain a candidate virtual image, and in response to the user selecting the candidate virtual image as the target virtual image, obtains a feature representation that conforms to the feature distribution of the target virtual image as a new image feature, and generates a final virtual image based on the new image feature and text features. That is to say, in the process of generating a virtual image, on the one hand, it is not necessary to rely on computer graphics modeling, and on the other hand, because several image features are obtained in the initial stage to generate candidate virtual images that conform to various feature distributions, the image is generated again in the subsequent stage through the candidate virtual image selected by the user, and then the final virtual image generated based on it can meet the user's personalized expectations as much as possible. Therefore, on the basis of meeting the personalized virtual image generation needs, it is possible to reduce the time and cost of virtual image generation and reduce the requirements for hardware equipment.
[0051] See also Figure 3 , Figure 3 This is a flow chart of an embodiment of training an image generation model. Specifically, it may include the following steps:
[0052] Step S31: Select a sample virtual image as a sample target image, and use the sample description text annotated with the sample target image as the sample target text.
[0053] In the disclosed embodiment, as described above, the candidate virtual image is generated by an image generation model, which is trained based on a number of sample virtual images, and the sample virtual images are annotated with sample description texts. In this case, any sample virtual image can be selected as a sample target image, and the sample description text annotated with the sample target image is used as the sample target text.
[0054] Step S32: extracting target image features based on the sample target image, and extracting target text features based on the sample target text.
[0055] Specifically, the specific process of extracting the target text feature can refer to the relevant description of extracting the text feature of the description text in the aforementioned disclosed embodiment, which will not be repeated here. In addition, the specific process of extracting the target image feature can refer to the relevant description of extracting the sample image feature in the aforementioned disclosed embodiment, which will not be repeated here. For the convenience of description, the target text feature can be recorded as Tc.
[0056] Step S33: obtaining positive image features that have the same feature distribution as the target image features, and obtaining negative image features that do not have the same feature distribution as the target image features.
[0057] Specifically, as mentioned above, the candidate feature distribution is obtained based on a number of sample virtual images, and can be obtained through principal component analysis, feature clustering and other processes. For details, please refer to the relevant description in the aforementioned public embodiment, which will not be repeated here. In this case, the positive image feature can be obtained by sampling the candidate feature distribution that meets the target image feature, and the negative image feature can be obtained by sampling the candidate feature distribution that does not meet the target image feature.
[0058] In an implementation scenario, the candidate feature distribution corresponding to the cluster of the sample target image from which the target image features are extracted can be used as the positive feature distribution, and the positive image features can be obtained by sampling from the positive feature distribution. Similarly, any candidate feature distribution other than the positive feature distribution among several candidate feature distributions can be used as the negative feature distribution, and the negative image features can be obtained by sampling from the negative feature distribution.
[0059] In an implementation scenario, for example, please refer to Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a process of training an image generation model according to an embodiment of the present invention. Figure 4 As shown in the figure, after the principal component analysis and feature clustering of several sample virtual images, five candidate feature distributions can be obtained. Then, for the selected sample target image, the candidate feature distribution corresponding to its cluster can be selected as the positive feature distribution, and other candidate feature distributions can be selected as the negative feature distribution. On this basis, the target image feature F can be extracted based on the sample target image. anchor , and sample the positive image feature F from the positive feature distribution pos , and the negative image feature F is obtained by sampling from the negative feature distribution neg In particular, the positive image features can be obtained by sampling from the [-2σ, 2σ] interval of the positive feature distribution, and the negative image features can be obtained by sampling from the [-2σ, 2σ] interval of the negative feature distribution. For the specific sampling process, please refer to the relevant description of obtaining image features by sampling from each candidate feature distribution in the aforementioned disclosed embodiment, which will not be repeated here.
[0060] Step S34: obtaining a synthetic target image, a synthetic positive image and a synthetic negative image based on the target image feature, one of the positive image feature and the negative image feature and the target text feature respectively.
[0061] Specifically, we can use the target image feature F anchor and the target text feature Tc, synthesize the synthetic target image, and based on the positive image feature F pos and the target text feature Tc, synthesized to obtain the synthetic positive image, and the negative image feature F negand the target text feature Tc, and synthesize them to obtain a synthetic negative image. anchor , positive example image feature F pos and negative image feature F neg The tensor of 3*1*(S+1)*D dimensions is formed by splicing, denoted as B, as the input of the mapping network. After a series of fully connected layers connected in sequence in the mapping network, the tensor W can be obtained, and after a series of fully connected layers connected in parallel in the mapping network, several (e.g., 18) new feature tensors can be obtained, which can be denoted as [W1, W2, ..., W18]. On this basis, the group of tensors can be input into different layers of the generation network respectively to control the generation of image content at different levels. For the specific process, please refer to the related description of generating candidate virtual images based on the image features and the text features of the description text in the aforementioned public embodiment, which will not be repeated here. For the convenience of description, the synthesized target image can be denoted as I anchor , the synthetic positive image is recorded as I pos , the synthetic negative image is recorded as I neg .
[0062] Step S35: adjusting the network parameters of the image generation model based on the synthesized target image, the synthesized positive image and the synthesized negative image.
[0063] In an implementation scenario, in order to constrain the synthetic target image, the synthetic positive image, and the synthetic negative image to have a certain degree of similarity with the sample target text at the semantic level, a first loss can be obtained based on the semantic similarity between the sample target text and the synthetic target image, the synthetic positive image, and the synthetic negative image, and the network parameters can be adjusted based on the first loss. The above method, based on the semantic similarity between the sample target text and the synthetic target image, the synthetic positive image, and the synthetic negative image, respectively, obtains the first loss, and adjusts the network parameters based on the first loss, which can ensure that the synthetic target image, the synthetic positive image, and the synthetic negative image have consistent semantics with the sample target text as much as possible, which helps to improve the semantic understanding of the input text by the image generation model, and avoids the image generation model from generating a virtual image that has no semantic association with the input text as much as possible, which is conducive to improving the model performance of the image generation model.
[0064] In a specific implementation scenario, in order to obtain semantic features of two different modal data, the sample target text and the synthesized virtual image, a pre-trained network for extracting semantic features of different modal data can be pre-trained, such as a CLIP network. The specific structure and training process of the pre-trained network can refer to the technical details of the CLIP network, which will not be described here. It should be noted that the pre-trained network can include an image encoder (such as Figure 4 face feature extractor) and text encoder (such as Figure 4 On this basis, the image encoder can be used to extract the semantic features of the synthetic target image, the synthetic positive image, and the synthetic negative image, which are respectively denoted as IC anchor 、IC pos 、IC neg At the same time, the text encoder can be used to extract the semantic features of the sample target text, denoted as TC anchor .
[0065] In a specific implementation scenario, during the training process, the semantic features of the synthetic target image, the synthetic positive image, and the synthetic negative image can be constrained to be IC anchor 、IC pos 、IC neg The semantic features of the sample target text are as close as possible to TC anchor In this case, the first loss L sim It can be expressed as:
[0066] L sim =3-[cos(TC anchor ,IC anchor )+cos(TC anchor ,IC pos )+cos(TC anchor ,IC neg )]……(1)
[0067] In the above formula (1), cos() represents cosine similarity. For the specific calculation formula, please refer to the technical details of cosine similarity, which will not be repeated here.
[0068] In one implementation scenario, in order to further constrain the differences between different synthesized virtual images, before adjusting the network parameters based on the first loss, a second loss can be further obtained based on the semantic difference between the synthesized target image and the synthesized positive image, and the semantic difference between the synthesized target image and the synthesized negative image, so that the network parameters can be adjusted based on the first loss and the second loss. The above method obtains the second loss based on the semantic difference between the synthesized target image and the synthesized positive image, and the semantic difference between the synthesized target image and the synthesized negative image, and adjusts the network parameters based on the first loss and the second loss, which can improve the discrimination between virtual images generated by image features that conform to different feature distributions as much as possible, which is conducive to improving the model performance of the image generation model.
[0069] In a specific implementation scenario, the semantic difference between the synthesized target image and the synthesized positive image, as well as the semantic difference between the synthesized target image and the synthesized negative image can be measured by, for example, the L2 norm. On this basis, the triplet loss can be used to process the above two semantic differences so as to bring the semantic features of the synthesized target image and the synthesized positive image closer together, and to push the semantic features of the synthesized target image and the synthesized negative image further apart.
[0070] In a specific implementation scenario, the second loss L triplet It can be expressed as:
[0071] L triplet =||IC anchor -IC pos ||2 2 —||IC anchor -IC neg ||2 2 +margin……(2)
[0072] In the above formula (2), ||||2 represents the L2 norm, margin represents the metric spacing, and the specific values are not limited.
[0073] In an implementation scenario, in addition to the first loss and the second loss, the adversarial loss can also be used to constrain the distribution of the generated virtual image so that it is as consistent as possible with the real distribution. Specifically, the adversarial loss L adv It can be expressed as:
[0074] L adv =E z~Pz [f w (G(z))]-E x~pr [f w (x)]……(3)
[0075] In the above formula (3), G(z) represents the virtual image generated by the image generation model (e.g., the aforementioned synthetic target image, synthetic positive image, synthetic negative image), x represents the real virtual image (e.g., the aforementioned sample target image), f w represents the distribution function, and E represents the expected value of the distribution function. For the specific meaning of adversarial loss, please refer to the technical details of the generative adversarial network, which will not be repeated here.
[0076] In an implementation scenario, in addition to the first loss, the second loss, and the adversarial loss, feature matching loss can also be used to constrain the model training to make it more stable. Specifically, feature matching loss L FM It can be expressed as:
[0077] L FM =∑ Ll=0 ||D l (I fake ),D l (I real )||2 2 ……(4)
[0078] In the above formula (4), I fake represents a virtual image generated by the image generation model (e.g., the aforementioned synthetic target image, synthetic positive image, synthetic negative image), I real represents the real virtual image (e.g., the sample target image mentioned above), l represents the lth network layer in the discriminator D, then D l (I fake ) indicates I fake The features output after being processed by the lth network layer in the discriminator D, D l (I real ) indicates I real The features output after being processed by the lth network layer in the discriminator D. In addition, ||||2 represents the L2 norm.
[0079] In one implementation scenario, the first loss, the second loss, the adversarial loss, and the feature matching loss can be weighted to obtain a weighted loss, and the network parameters can be adjusted based on the weighted loss. For example, an optimization method such as gradient descent can be used to adjust the network parameters of the image generation model based on the weighted loss. In addition, the weighted loss L total It can be expressed as:
[0080] L total =α1*L sim +α2*L adv +α3*L FM +α1*L triplet ……(5)
[0081] The above scheme selects a sample virtual image as a sample target image, and uses the sample description text annotated with the sample target image as the sample target text, extracts target image features based on the sample target image, and extracts target text features based on the sample target text. On this basis, the positive image features that conform to the same feature distribution as the target image features are obtained, and the negative image features that do not conform to the same feature distribution as the target image features are obtained, and based on one of the target image features, the positive image features, and the negative image features and the target text features, respectively, a synthetic target image, a synthetic positive image, and a synthetic negative image are obtained, and then based on the synthetic target image, the synthetic positive image, and the synthetic negative image, the network parameters of the image generation model are adjusted, which can improve the model performance of the image generation model as much as possible.
[0082] See also Figure 5 , Figure 5 It is a schematic diagram of a framework of an embodiment of a virtual image generation device 50 of the present application. The virtual image generation device 50 includes: an acquisition module 51, a generation module 52, a response module 53 and a loop module 54. The acquisition module 51 is used to obtain the user's description text of the desired virtual image and obtain a number of image features; the generation module 52 is used to generate an image based on each image feature and the text features of the description text, and obtain a candidate virtual image; the response module 53 is used to respond to the user selecting the candidate virtual image as the target virtual image, and obtain a feature representation that meets the feature distribution of the target virtual image as a new image feature; the loop module 54 is used to generate the final virtual image based on the new image features and text features.
[0083] In the above scheme, during the process of generating a virtual image, on the one hand, it does not need to rely on computer graphics modeling, and on the other hand, because several image features are obtained in the initial stage to generate candidate virtual images that meet the distribution of various features, the image is generated again in the subsequent stage through the candidate virtual image selected by the user, and the final virtual image generated based on it can meet the user's personalized expectations as much as possible. Therefore, on the basis of meeting the personalized virtual image generation needs, the time and cost of virtual image generation can be reduced, and the requirements for hardware equipment can be reduced.
[0084] In some disclosed embodiments, the candidate virtual images are generated by an image generation model, which is trained based on a number of sample virtual images, and the sample virtual images are annotated with sample description text.
[0085] Therefore, the candidate virtual images are generated by an image generation model. The image generation model is trained based on a number of sample virtual images. The sample virtual images are annotated with sample description texts, which can improve the efficiency of generating virtual images.
[0086] In some disclosed embodiments, the virtual image generation device 50 includes a sample selection module for selecting a sample virtual image as a sample target image, and using a sample description text annotated with the sample target image as a sample target text; the virtual image generation device 50 includes a sample extraction module for extracting target image features based on the sample target image, and extracting target text features based on the sample target text; the virtual image generation device 50 includes a sample sampling module for obtaining positive image features that conform to the same feature distribution as the target image features, and obtaining negative image features that do not conform to the same feature distribution as the target image features; the virtual image generation device 50 includes a sample synthesis module for obtaining a synthetic target image, a synthetic positive image, and a synthetic negative image based on one of the target image features, the positive image features, and the negative image features and the target text features, respectively; the virtual image generation device 50 includes a parameter adjustment module for adjusting the network parameters of the image generation model based on the synthetic target image, the synthetic positive image, and the synthetic negative image.
[0087] Therefore, the sample virtual image is selected as the sample target image, and the sample description text annotated by the sample target image is used as the sample target text. The target image features are extracted based on the sample target image, and the target text features are extracted based on the sample target text. On this basis, the positive image features that conform to the same feature distribution as the target image features are obtained, and the negative image features that do not conform to the same feature distribution as the target image features are obtained. Based on the target image features, the positive image features, and the negative image features and the target text features, a synthetic target image, a synthetic positive image, and a synthetic negative image are obtained. Based on the synthetic target image, the synthetic positive image, and the synthetic negative image, the network parameters of the image generation model are adjusted to improve the model performance of the image generation model as much as possible.
[0088] In some disclosed embodiments, the parameter adjustment module includes a first metric submodule, which is used to obtain a first loss based on the semantic similarity between the sample target text and the synthetic target image, the synthetic positive image, and the synthetic negative image, respectively; the parameter adjustment module includes a network parameter adjustment submodule, which is used to adjust the network parameters based on the first loss.
[0089] Therefore, based on the semantic similarity between the sample target text and the synthetic target image, the synthetic positive image, and the synthetic negative image, the first loss is obtained, and based on the first loss, the network parameters are adjusted to ensure that the synthetic target image, the synthetic positive image, and the synthetic negative image have consistent semantics with the sample target text as much as possible, which helps to improve the image generation model's semantic understanding of the input text, and avoids the image generation model from generating virtual images that have no semantic association with the input text as much as possible, which is conducive to improving the model performance of the image generation model.
[0090] In some disclosed embodiments, the parameter adjustment module includes a second metric submodule for obtaining a second loss based on the semantic difference between the synthesized target image and the synthesized positive image, and the semantic difference between the synthesized target image and the synthesized negative image; and the network parameter adjustment submodule is specifically used to adjust the network parameters based on the first loss and the second loss.
[0091] Therefore, based on the semantic difference between the synthesized target image and the synthesized positive image, as well as the semantic difference between the synthesized target image and the synthesized negative image, the second loss is obtained, and based on the first loss and the second loss, the network parameters are adjusted. This can improve the distinction between virtual images generated by image features that conform to different feature distributions as much as possible, which is conducive to improving the model performance of the image generation model.
[0092] In some disclosed embodiments, several image features are sampled from several candidate feature distributions, the several candidate feature distributions respectively represent feature distributions of different types of virtual images, and the candidate feature distributions are obtained based on several sample virtual images.
[0093] Therefore, several image features are obtained by sampling from several candidate feature distributions, and the several candidate feature distributions respectively represent the feature distributions of different types of virtual images, and the candidate feature distributions are obtained based on several sample virtual images, so the diversity of image features can be ensured as much as possible.
[0094] In some disclosed embodiments, the virtual image generating device 50 includes a principal component analysis module for extracting sample image features of each sample virtual image based on principal component analysis; the virtual image generating device 50 includes a central feature acquisition module for obtaining central features of a preset number of clustering centers based on a preset number of principal components obtained by performing principal component analysis; the virtual image generating device 50 includes a feature clustering module for clustering sample image features based on each central feature to obtain candidate feature distributions that respectively characterize different types of virtual images.
[0095] Therefore, the sample virtual image is first extracted based on principal component analysis, and then clustered through a preset number of principal components to obtain the candidate feature distribution that characterizes virtual images of different virtual types. Therefore, feature clustering can be performed in an unsupervised manner, which helps to improve the accuracy of the candidate feature distribution.
[0096] In some disclosed embodiments, the principal component analysis module includes an image encoding submodule, which is used to encode each sample virtual image to obtain sample encoding features; the principal component analysis module includes a feature projection submodule, which is used to reproject the sample encoding features to an orthogonal space based on principal component analysis to obtain sample image features.
[0097] Therefore, each sample virtual image is encoded to obtain the sample encoding features, and the sample encoding features are reprojected to the orthogonal space based on principal component analysis to obtain the sample image features, which can make each cluster center more distinguishable and thus improve the accuracy of the candidate feature distribution.
[0098] In some disclosed embodiments, the response module includes a first response submodule for obtaining new image features based on target feature interpolation, and the response module includes a second response submodule for obtaining new image features by sampling the target feature distribution; wherein the target feature is an image feature of a target virtual image generated, and the plurality of image features are respectively sampled from a plurality of candidate feature distributions, and the plurality of candidate feature distributions respectively represent feature distributions of different types of virtual images, and the target feature distribution is a candidate feature distribution of the target feature obtained by sampling.
[0099] Therefore, new image features are obtained based on interpolation of target features, and the target features are image features for generating a target virtual image, so that a feature representation that conforms to the feature distribution of the target virtual image can be directly obtained through an interpolation algorithm, thereby improving the convenience of generating new image features; and new image features are obtained by sampling the target feature distribution, so that a feature representation that conforms to the feature distribution of the target virtual image can be obtained through resampling, thereby improving the accuracy of generating new image features.
[0100] In some disclosed embodiments, the loop module is specifically used to re-execute the steps of generating candidate virtual images based on each image feature and the text features of the descriptive text and subsequent steps based on the new image features, until the user selects at least one candidate virtual image as the final virtual image.
[0101] Therefore, based on the new image features, the steps of generating candidate virtual images based on each image feature and the text features of the descriptive text and subsequent steps are re-executed until the user selects at least one candidate virtual image as the final virtual image. Therefore, by generating virtual images through multiple iterations, the generated virtual image can be infinitely close to the user's expected virtual image. This is different from the method of pre-constructing a database containing a limited number of candidate virtual images and selecting the final virtual image from the database. By re-acquiring new image features during each iteration, a completely new virtual image can be generated during each iteration, thereby maximizing the richness of the candidate virtual images and increasing the possibility of the user selecting a final virtual image that is infinitely close to the expected virtual image, which helps to greatly improve user satisfaction.
[0102] See also Figure 6 , Figure 61 is a schematic diagram of a framework of an embodiment of an electronic device 60 of the present application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other, the memory 61 stores program instructions, and the processor 62 is used to execute the program instructions to implement the steps in any of the above-mentioned virtual image generation method embodiments. Specifically, the electronic device 60 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.
[0103] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in any of the above-mentioned virtual image generation method embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.
[0104] In the above scheme, during the process of generating a virtual image, on the one hand, it does not need to rely on computer graphics modeling, and on the other hand, because several image features are obtained in the initial stage to generate candidate virtual images that meet the distribution of various features, the image is generated again in the subsequent stage through the candidate virtual image selected by the user, and the final virtual image generated based on it can meet the user's personalized expectations as much as possible. Therefore, on the basis of meeting the personalized virtual image generation needs, the time and cost of virtual image generation can be reduced, and the requirements for hardware equipment can be reduced.
[0105] See also Figure 7 , Figure 7 Schematic diagram of a framework of an embodiment of a computer-readable storage medium 70 of the present application. The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor, and the program instructions 71 are used to implement the steps in any of the above-mentioned embodiments of the virtual image generation method.
[0106] In the above scheme, during the process of generating a virtual image, on the one hand, it does not need to rely on computer graphics modeling, and on the other hand, because several image features are obtained in the initial stage to generate candidate virtual images that meet the distribution of various features, the image is generated again in the subsequent stage through the candidate virtual image selected by the user, and the final virtual image generated based on it can meet the user's personalized expectations as much as possible. Therefore, on the basis of meeting the personalized virtual image generation needs, the time and cost of virtual image generation can be reduced, and the requirements for hardware equipment can be reduced.
[0107] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0108] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0109] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0111] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
[0113] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.
Claims
1. A method for generating a virtual image, characterized in that: include: Obtaining a user's description text of a desired virtual image and obtaining a number of image features; Based on each of the image features and the text features of the description text, image generation is performed to obtain a candidate virtual image; In response to the user selecting the candidate virtual image as the target virtual image, a new image feature is obtained based on the target feature interpolation, or a new image feature is obtained by sampling the target feature distribution; wherein the target feature is the image feature generated to obtain the target virtual image, the plurality of image features are respectively sampled from a plurality of candidate feature distributions, the plurality of candidate feature distributions respectively represent feature distributions of different types of virtual images, and the target feature distribution is the candidate feature distribution obtained by sampling the target feature; The final virtual image is generated based on the new image features and the text features, specifically comprising: based on the new image features and the text features, generating a new candidate virtual image as the final virtual image; or, based on the new image features, re-executing the step of generating an image based on each of the image features and the text features of the descriptive text, obtaining a candidate virtual image and iterating until the user selects at least one of the candidate virtual images as the final virtual image.
2. The method according to claim 1, characterized in that: The candidate virtual image is generated by an image generation model, and the image generation model is trained based on a number of sample virtual images, and the sample virtual images are annotated with sample description text.
3. The method according to claim 2, characterized in that The training steps of the image generation model include: Select the sample virtual image as a sample target image, and use the sample description text annotated with the sample target image as a sample target text; Extracting target image features based on the sample target image, and extracting target text features based on the sample target text; Acquire positive image features that have the same feature distribution as the target image features, and acquire negative image features that do not have the same feature distribution as the target image features; Based on the target image feature, one of the positive example image feature and the negative example image feature and the target text feature, respectively, a synthetic target image, a synthetic positive example image and a synthetic negative example image are obtained; Based on the synthesized target image, the synthesized positive image and the synthesized negative image, the network parameters of the image generation model are adjusted.
4. The method according to claim 3, characterized in that The adjusting the network parameters of the image generation model based on the synthesized target image, the synthesized positive image and the synthesized negative image includes: Obtaining a first loss based on the semantic similarities between the sample target text and the synthetic target image, the synthetic positive example image, and the synthetic negative example image respectively; Based on the first loss, the network parameters are adjusted.
5. The method according to claim 4, characterized in that Before adjusting the network parameters based on the first loss, the method further includes: Obtaining a second loss based on a semantic difference between the synthetic target image and the synthetic positive example image, and a semantic difference between the synthetic target image and the synthetic negative example image; The adjusting the network parameter based on the first loss includes: The network parameters are adjusted based on the first loss and the second loss.
6. The method according to claim 1, characterized in that The candidate feature distribution is obtained based on a number of sample virtual images.
7. The method according to claim 6, characterized in that The step of obtaining the candidate feature distribution includes: Extracting sample image features of each of the sample virtual images based on principal component analysis; Based on a preset number of principal components obtained by performing the principal component analysis, obtaining central features of the preset number of cluster centers; The sample image features are clustered based on each of the central features to obtain candidate feature distributions that respectively characterize different types of virtual images.
8. The method according to claim 7, characterized in that The extracting of sample image features of each sample virtual image based on principal component analysis includes: Encoding each of the sample virtual images to obtain sample encoding features; The sample encoding features are reprojected into an orthogonal space based on the principal component analysis to obtain the sample image features.
9. A virtual image generating device, characterized in that: include: An acquisition module, used to acquire a user's description text of a desired virtual image and acquire a number of image features; A generation module, used for performing image generation based on each of the image features and the text features of the description text to obtain a candidate virtual image; A response module, configured to obtain a new image feature based on target feature interpolation or obtain a new image feature by sampling a target feature distribution in response to the user selecting the candidate virtual image as the target virtual image; wherein the target feature is an image feature generated to obtain the target virtual image, the plurality of image features are respectively obtained by sampling from a plurality of candidate feature distributions, the plurality of candidate feature distributions respectively represent feature distributions of different types of virtual images, and the target feature distribution is a candidate feature distribution obtained by sampling the target feature; A loop module is used to generate a final virtual image based on the new image features and the text features, and is specifically used to: generate a new candidate virtual image as the final virtual image based on the new image features and the text features; or, based on the new image features, re-execute the steps of generating an image based on each of the image features and the text features of the descriptive text, obtain a candidate virtual image, and iterate until the user selects at least one of the candidate virtual images as the final virtual image.
10. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the virtual image generation method according to any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the virtual image generation method described in any one of claims 1 to 8.
