Method and device for generating text image

Abstract

The present application discloses a method and device for generating text images in the field of artificial intelligence (AI). The method comprises: obtaining n text images to be processed, determining n prediction latent vectors corresponding to the n text images to be processed in a generative adversarial network (GAN) model, determining a target latent vector based on the n prediction latent vectors, and generating a target text image from the target latent vector through the GAN model. The method of the present application can generate text images of a specific style by using a deep learning method.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a method and device for generating text images. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] With the rapid development of artificial intelligence technology, more and more people are beginning to pay attention to this "new" research field: deep learning. Deep learning has not only surpassed traditional methods in image classification, facial recognition and other tasks, but also surpassed human level. The three main factors affecting deep learning are: massive training data, powerful computing architecture and relevant progress in the academic field. In the process of using deep learning to solve problems, the most common obstacle is the massive data required in the process of training the model. In the process of building some models, such as text recognition models, a large number of text images are required to realize model training and testing. However, there are still many problems with the traditional way of obtaining text images. For example, it is difficult to obtain pictures by real shooting, the coverage area is small, and the human resource cost is high; for example, the quality of pictures obtained by web crawling is uneven, and there may be copyright risks; for example, the cost of obtaining images by purchasing third-party data is high and the cycle is long.

[0004] At present, text images can also be generated through neural network models, which can solve the above problems. For example, rich text images can be generated using traditional generative adversarial networks (GANs). In the process of model training and testing, text images of specific styles are often required, for example, text images in real scenes are required. However, traditional GANs cannot control the style of the generated text images, and the text in the generated text images may not have reasonable semantics, which affects the subsequent training and testing of the model.

[0005] Therefore, how to obtain text images with a specific style has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present application provides a method and device for generating a text image, which can generate a text image of a specific style.

[0007] In a first aspect, a method for generating a text image is provided, the method comprising: obtaining n text images to be processed, wherein n is an integer greater than or equal to 2; determining n prediction latent vectors corresponding to the n text images to be processed in a generative adversarial network (GAN) model; determining a target latent vector based on the n prediction latent vectors; and generating a target text image from the target latent vector through the GAN model.

[0008] Among them, a text image refers to an image containing text.

[0009] Exemplarily, the GAN model may be a StyleGAN model or a StyleGAN2 model, etc.

[0010] In the embodiment of the present application, the latent vector may also be referred to as a hidden code or a latent vector code or Latent.

[0011] Exemplarily, the text image to be processed may include a content text image and a style text image. The target text image may be a fusion of the content text image and the style text image. Specifically, the text content in the content text image and the image style in the style text image are fused to generate a target text image having the text content of the content text image and the image style of the style text image.

[0012] A text image used to provide text content in a target text image can be considered as a content text image. A text image used to provide image style in a target text image can be considered as a style text image.

[0013] The content text image may include one text image or multiple text images, and the style text image may include one text image or multiple text images.

[0014] Determining the predicted latent vector corresponding to the text image to be processed in the GAN model can also be understood as embedding the text image to be processed into the latent vector space of the GAN model.

[0015] Exemplarily, n predicted latent vectors may be processed according to how the latent vectors control the generated text image in the GAN model to obtain a target latent vector.

[0016] Among them, the way in which the latent vector controls the generated text image in the GAN model refers to the influence of the elements in the latent vector on the image features in the generated text image. The influence of the elements in the latent vector on the image features in the generated text image can also be understood as the relationship between the elements in the latent vector and the image features.

[0017] Different elements in a latent vector encoding can control different features of the generated image. For example, one element in a latent vector encoding can control a feature of the generated text image. Alternatively, multiple elements in a latent vector encoding can jointly control a feature of the generated text image. In other words, the feature of the generated text image can be controlled by one element in a latent vector encoding, or by multiple elements.

[0018] In the embodiment of the present application, the target latent vector is determined by the predicted latent vector corresponding to the text image to be processed, and the characteristics of the target text image can be controlled by the target latent vector. Different characteristics of the image determine the different styles of the image. Through the scheme of the embodiment of the present application, the target latent vector can be flexibly adjusted according to the required style to generate a text image with a specific style to meet the application requirements.

[0019] In addition, since the GAN model does not have semantic perception capabilities, the text in the text image generated directly by the GAN model may not have reasonable semantics. The scheme of the embodiment of the present application can bypass the semantic processing process, and by determining the predicted latent vector corresponding to the text image to be processed, the text image to be processed can retain the correct semantics of the text. For example, the predicted latent vector corresponding to the content text image can generate a text image close to the content text image, and the shape of the text in the content text image can be retained, which will not affect the semantics of the image. In this way, when the GAN model is not aware of semantics, by retaining the elements that control the shape of the text in the content text image in the target latent vector, the target latent vector can be used to generate a target text image with reasonable semantics.

[0020] In combination with the first aspect, in certain implementations of the first aspect, determining the target latent vector based on the n prediction latent vectors includes: cross-mixing the n prediction latent vectors to obtain the target latent vector, wherein the target latent vector includes at least one element in each of the n prediction latent vectors, and at least one element in each of the prediction latent vectors is used to control the target image features of the target text image.

[0021] Specifically, the target element for controlling the target image feature can be determined according to the relationship between the elements in the latent vector and the image feature, and then the n predicted latent vectors can be cross-mixed according to the target element.

[0022] For example, the target image feature includes the feature P of the first text image and the feature Q of the second text image, and the target element is determined according to the relationship between the element in the latent vector and the image feature. The target element includes the element P in the prediction latent vector corresponding to the first text image and the element Q in the prediction latent vector corresponding to the second text image, and the element Q in the prediction latent vector corresponding to the first text image can be replaced with the element Q in the prediction latent vector corresponding to the second text image, and the corresponding prediction latent vector of the replaced first text image is used as the target latent vector.

[0023] The element P can be understood as the element at position P in the latent vector. The element Q can be understood as the element at position Q in the latent vector.

[0024] According to the scheme of the embodiment of the present application, the required elements in the target latent vector can be determined according to the required image features in the target text image, or in other words, the mixing method of the predicted latent vector can be determined to obtain the target latent vector that can generate the target text image, which can further improve the controllability of the style of the generated target text image and obtain the desired target text image.

[0025] In combination with the first aspect, in some implementations of the first aspect, the target image feature includes the text shape, background or text font.

[0026] According to the scheme of the embodiment of the present application, the target text image can be generated by changing the text shape, background or font and other features of the text image to be processed. Compared with the scheme that cannot change specific features such as text font, this scheme greatly improves the diversity of the generated target text images.

[0027] In combination with the first aspect, in certain implementations of the first aspect, determining the n predicted latent vectors corresponding to the n text images to be processed in the GAN model includes: inputting the n images to be processed into a latent vector encoding model for processing to obtain n predicted latent vectors corresponding to the n text images to be processed in the GAN model; wherein the latent vector encoding model is obtained by training the latent vector encoding model with a first candidate text image as the input of the latent vector encoding model and with a first latent vector corresponding to the first candidate text image in the GAN model as the target output of the latent vector encoding model, and the first latent vector is used to generate the first candidate text image in the GAN.

[0028] The input of the latent vector encoding model is a text image, and the output is a predicted latent vector corresponding to the text image in the GAN model. That is to say, the first candidate text image generated by the GAN model is input into the latent vector encoding model, so that the output predicted latent vector is as close to the first latent vector as possible, or the difference between the output predicted latent vector and the first latent vector is minimized.

[0029] The latent vector encoding model may be a neural network model, that is, the mapping relationship from the text image space to the latent vector space is learned through the neural network model, or the mapping relationship from the first candidate text image to the first latent vector is learned.

[0030] According to the solution of the embodiment of the present application, the training data of the latent vector coding model is obtained through the GAN model, and the latent vector coding model is trained based on the training data, and the predicted latent vector of the text image to be processed can be obtained by using the trained latent vector coding model. In this way, the predicted latent vector can be determined by the trained latent vector coding model.

[0031] In combination with the first aspect, in certain implementations of the first aspect, determining the n predicted latent vectors corresponding to the n text images to be processed in the GAN model includes: step one: sampling a second latent vector using the GAN model; step two: generating a second candidate text image based on the second latent vector; step three: determining the difference between the second candidate text image and the first text image, wherein the first text image is one of the n text images to be processed; step four: updating the second latent vector, and using the updated second latent vector as the second latent vector in step two; repeating steps two to four until a termination condition is met, and using the updated second latent vector as the predicted latent vector corresponding to the first text image in the GAN model, wherein the termination condition includes at least one of the following: the difference between the second candidate text image and the first text image is less than or equal to a first threshold, or the number of repetitions is greater than or equal to a second threshold.

[0032] Exemplarily, the difference between two text images can be determined by feature distance. The smaller the feature distance between two text images, the smaller the difference between the two image texts. The feature distance between two text images refers to the difference between the feature values ​​of the two text images.

[0033] It should be noted that only the first text image is taken as an example here, and other text images among the n text images to be processed can perform the same operation as the first text image to obtain the prediction latent vectors corresponding to the n text images to be processed.

[0034] According to the solution of the embodiment of the present application, the prediction latent vector corresponding to the text image to be processed is determined by minimizing the difference between the text image to be processed and the second candidate text image.

[0035] In combination with the first aspect, in certain implementations of the first aspect, determining the degree of difference between the second candidate text image and the first text image includes: performing feature extraction on the second candidate text image to obtain a feature value of the second candidate text image; performing feature extraction on the first text image to obtain a feature value of the first text image; and using the difference between the feature value of the second candidate text image and the feature value of the first text image as the degree of difference between the second candidate text image and the first text image.

[0036] Specifically, the image feature extraction model can be used to extract features from the image to obtain the feature value of the image. For example, the image feature extraction model can be a visual geometry group network (VGG-16) network model.

[0037] In a second aspect, a device for generating a text image is provided, the device comprising: an acquisition unit, used to acquire an image to be processed, wherein n is an integer greater than or equal to 2; a processing unit, used to: determine n predicted latent vectors corresponding to the n text images to be processed in a generative adversarial network (GAN) model; determine a target latent vector based on the n predicted latent vectors; and generate a target text image from the target latent vector through the GAN model.

[0038] In combination with the second aspect, in certain implementations of the second aspect, the processing unit is specifically used to: cross-mix the n prediction latent vectors to obtain the target latent vector, wherein the target latent vector includes at least one element in each of the n prediction latent vectors, and at least one element in each of the prediction latent vectors is used to control the target image features of the target text image.

[0039] In combination with the second aspect, in certain implementations of the second aspect, the target image feature includes the text shape, background, or text font.

[0040] In combination with the second aspect, in certain implementations of the second aspect, the processing unit is specifically used to: input the n images to be processed into a latent vector encoding model for processing, and obtain n predicted latent vectors corresponding to the n text images to be processed in the GAN model; wherein the latent vector encoding model is obtained by training the latent vector encoding model with a first candidate text image as the input of the latent vector encoding model and with a first latent vector corresponding to the first candidate text image in the GAN model as the target output of the latent vector encoding model, and the first latent vector is used to generate the first candidate text image in the GAN.

[0041] In combination with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to: Step 1: Sample a second latent vector using the GAN model; Step 2: Generate a second candidate text image based on the second latent vector; Step 3: Determine the difference between the second candidate text image and the first text image, where the first text image is one of the n text images to be processed; Step 4: Update the second latent vector, and use the updated second latent vector as the second latent vector in step 2; Repeat steps 2 to 4 until a termination condition is met, and use the updated second latent vector as the predicted latent vector corresponding to the first text image in the GAN model, wherein the termination condition includes at least one of the following: the difference between the second candidate text image and the first text image is less than or equal to a first threshold, or the number of repetitions is greater than or equal to a second threshold.

[0042] In combination with the second aspect, in certain implementations of the second aspect, the processing unit is specifically used to: perform feature extraction on the second candidate text image to obtain a feature value of the second candidate text image; perform feature extraction on the first text image to obtain a feature value of the first text image; and use the difference between the feature value of the second candidate text image and the feature value of the first text image as the degree of difference between the second candidate text image and the first text image.

[0043] It should be understood that the expansion, limitation, explanation and description of the relevant contents in the above-mentioned first aspect also apply to the same contents in the second aspect.

[0044] In a third aspect, a device for generating a text image is provided, the device comprising: a memory for storing a program; a processor for executing the program stored in the memory, when the program stored in the memory is executed, the processor is used to execute the method in the first aspect and any one of the implementations of the first aspect.

[0045] The processor in the third aspect can be a central processing unit (CPU) or a combination of a CPU and a neural network processor, where the neural network processor can include a graphics processing unit (GPU), a neural network processing unit (NPU) and a tensor processing unit (TPU), etc. Among them, TPU is a dedicated integrated circuit for artificial intelligence accelerator fully customized by Google for machine learning.

[0046] According to a fourth aspect, a computer-readable medium is provided, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes a method for executing any one of the implementations of the first aspect.

[0047] According to a fifth aspect, a computer program product comprising instructions is provided. When the computer program product is run on a computer, the computer is enabled to execute a method in any one of the implementations of the first aspect.

[0048] In a sixth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes a method in any one of the implementation modes in the first aspect.

[0049] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in any one of the implementation methods in the first aspect.

[0050] The chip may specifically be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0051] In a seventh aspect, an electronic device is provided, which includes the device in any one of the second aspect or the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a system architecture provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of the structure of another convolutional neural network provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of the hardware structure of a chip provided in an embodiment of the present application;

[0057] Figure 6 This is the effect diagram of generating text images using the traditional GAN ​​model;

[0058] Figure 7 This is the effect diagram of using the neural style transfer algorithm to generate text images;

[0059] Figure 8 A schematic flowchart of a method for training a latent vector coding model provided in an embodiment of the present application;

[0060] Fig. 9 A schematic block diagram of a GAN model and a latent vector encoding model provided in an embodiment of the present application;

[0061] Fig.10 A schematic flow chart of a method for generating a text image provided in an embodiment of the present application;

[0062] Fig.11 A schematic flow chart of another method for generating a text image provided in an embodiment of the present application;

[0063] Fig.12 A schematic flow chart of a method for determining a prediction latent vector corresponding to a text image to be processed provided in an embodiment of the present application;

[0064] Fig.13 A schematic flow chart of another method for generating a text image provided in an embodiment of the present application;

[0065] Fig.14 is a schematic block diagram of a device for generating a text image provided by an embodiment of the present application;

[0066] Fig.15 It is a schematic block diagram of a training device for a latent vector coding model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] The method for generating text images provided in the embodiments of the present application can be applied in the field of deep learning and other scenarios that require text images of a specific style.

[0069] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application is shown. Figure 1 As shown, the deep learning process may include: data collection, data input, data exploration and preprocessing, training and testing algorithms or evaluation stages.

[0070] The data types collected may be data in various forms, such as photographed text images or scanned text images.

[0071] The input data can also be understood as the acquired data. Specifically, the collected data can be integrated to obtain a data set. The data set can be used in the subsequent training and testing algorithm process.

[0072] Furthermore, the data set may be subjected to data exploration and preprocessing. For example, data exploration and data preprocessing may include data cleaning, data conversion, or supplementing normative data. This step is optional, that is, the data set may be directly used in the subsequent training and testing algorithm process.

[0073] The training algorithm and the test algorithm may also be referred to as a training model and a test model, or may be referred to as modeling. The data used for modeling may be referred to as sample data. The sample data may include the aforementioned data set, or may include data obtained after data exploration and preprocessing of the data in the data set.

[0074] Specifically, the sample data includes training data and test data, wherein the training data is used to train the model, and the test data is used to test the trained model.

[0075] Evaluation and optimization refers to evaluating and optimizing based on the results of model testing to obtain a target model, which can then be deployed on the corresponding device. Specifically, the evaluation results can be fed back to the model training phase, and the model can continue to be trained until the target model is obtained. The target model can be a model that meets the target performance indicators. Alternatively, the target model can be a model that meets user needs.

[0076] Therefore, both the quantity and quality of sample data will affect the trained model. If the sample data only includes the data in the data set or only includes the data obtained after data exploration and preprocessing in the data set, then the data in the data set cannot meet the requirements of model training or model testing. For example, the trained model often fails to meet the target performance indicators when the data form is single or the data volume is insufficient.

[0077] The method for generating text images in the embodiment of the present application can be used for sample data expansion in deep learning. Using the expanded sample data to train a model or test a model makes it easier to obtain a target model, or to obtain a model with better performance, for example, a model with higher accuracy.

[0078] The method for generating text images provided in the embodiment of the present application can be used to expand sample data. For example, text images can be generated in the data exploration and preprocessing stage as training data or test data to expand sample data. Next, the model can be trained or tested using the expanded sample data to obtain a target model.

[0079] Specifically, the method for generating a text image in the embodiment of the present application can be applied to a text processing scenario in a text image, for example, text processing may include text recognition or text translation. The following is a brief introduction to the text processing scenario in a text image.

[0080] Text processing in text images:

[0081] When a user needs to process text in a text image, for example, convert the text in a text image into text data, the text in the text image can be recognized by a text recognition model, and then the text can be processed for user convenience.

[0082] By using the method for generating text images provided in the embodiment of the present application, a large number of text images can be obtained. The generated text images can then be used as sample data to train a text recognition model, thereby using the text recognition model to recognize text in the text images, and then process the text, and return the processing results to the user.

[0083] The method and apparatus provided in the embodiments of the present application can be used to expand the training database, such as Figure 2 The I / O interface 112 of the execution device 110 shown can send the text image processed / generated by the execution device 110 and the sample image input by the user as training data to the database 130, so that the training data maintained by the database 130 is richer, thereby providing richer training data for the training work of the training device 120.

[0084] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0085] (1) Neural Network

[0086] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:

[0087]

[0088] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0089] (2) Generative Adversarial Networks

[0090] Generative adversarial network (GAN) is a deep learning model. Generative adversarial network includes at least one generator and one discriminator. It produces better output by letting two neural networks learn in a game-like manner. These two neural networks can be deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: Taking the GAN that generates pictures as an example, suppose there are two networks, G (Generator) and D (Discriminator), where G is a network that generates pictures, randomly sampling from the latent space as input to generate pictures, recorded as G (z); D is a discriminator network, which is used to determine whether a picture is "real". Its input parameter is x, x represents a picture, and x is a real picture or the output of the generator network. The output D (x) represents the probability that x is a real picture. If it is 1, it means that it is 100% a real picture, and if it is 0, it means that it cannot be a real picture. In the process of training the generative adversarial network, the goal of the generative network G is to generate as realistic images as possible to deceive the discriminative network D, and the output results need to imitate the real samples in the training set as much as possible. The goal of the discriminative network D is to distinguish the images generated by G from the real images as much as possible. The two networks compete with each other and constantly adjust parameters. In this way, G and D constitute a dynamic "game" process, which is also the "confrontation" in the "generative adversarial network". The ultimate goal is to make the discriminative network unable to judge whether the output results of the generative network are real. The final result of the game is that, under ideal conditions, G can generate images G(z) that are "real", while D has difficulty in judging whether the images generated by G are real, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate images.

[0091] (3) Convolutional Neural Network

[0092] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. Convolution kernels can be formalized as matrices of random size, and convolution kernels can obtain reasonable weights through learning during the training process of convolutional neural networks. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0093] (4) Loss Function

[0094] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process of optimization before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0095] (5) Back propagation algorithm

[0096] The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0097] like Figure 2 As shown, the embodiment of the present application provides a system architecture 100. Figure 2 In the embodiment, the data collection device 160 is used to collect training data. For the method for generating a text image in the embodiment of the present application, the training data may include sample images.

[0098] After collecting the training data, the data collection device 160 stores the training data in the database 130 , and the training device 120 obtains the target model / rule 101 through training based on the training data maintained in the database 130 .

[0099] The following describes how the training device 120 obtains the target model / rule 101 based on the training data.

[0100] The above-mentioned target model / rule 101 can be used to implement the method for generating text images in the embodiment of the present application. The target model / rule 101 in the embodiment of the present application can be a neural network. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the training data maintained by the database 130, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0101] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 2 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server, a graphics workstation or a cloud. Figure 2In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include an image to be processed input by the client device in the embodiment of the present application.

[0102] Preprocessing module 113 and preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be processed) received by I / O interface 112. In the embodiment of the present application, preprocessing module 113 and preprocessing module 114 may be omitted (or only one of the preprocessing modules may be present), and the computing module 111 may be used directly to process the input data.

[0103] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0104] Finally, the I / O interface 112 returns the processing result, such as the generated text image, to the client device 140 to provide it to the user. Alternatively, the processing result, such as the generated text image, can be stored in the data storage system 150 to serve as training data for other models.

[0105] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0106] exist Figure 2In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0107] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0108] like Figure 2 As shown, the target model / rule 101 is obtained through training by the training device 120. The target model / rule 101 may include the GAN model, latent vector encoding model or image feature extraction model in the embodiment of the present application.

[0109] Since CNN is a very common neural network, for example, the neural network in the GAN model can be CNN. Figure 3 The focus is on a detailed introduction to the structure of CNN. As mentioned in the basic concept introduction above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture refers to an algorithm that updates the neural network model and performs multiple levels of learning at different levels of abstraction. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.

[0110] exist Figure 3In the figure, a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (wherein the pooling layer is optional), and a fully connected layer 230.

[0111] Convolutional layer / pooling layer 220:

[0112] Convolutional Layer:

[0113] like Figure 3 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0114] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0115] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise points in the image, etc. The multiple weight matrices have the same size (rows × columns), and the convolution feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted multiple convolution feature maps of the same size are then merged to form the output of the convolution operation.

[0116] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0117] When the convolutional neural network 200 has multiple convolutional layers, the convolutional layers (for example, 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0118] Pooling layer:

[0119] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 3Each layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0120] Fully connected layer 230:

[0121] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the fully connected layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the fully connected layer 230 may include multiple hidden layers (such as Figure 3 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0122] After the multiple hidden layers in the fully connected layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 3 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 3 The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0123] The structure of the convolutional neural network can also be Figure 4 As shown. Figure 4In FIG. 1 , a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (wherein the pooling layer is optional), and a fully connected layer 230. Figure 3 compared to, Figure 4 The multiple convolutional layers / pooling layers in the convolutional layer / pooling layer 220 are operated in parallel, and the features extracted respectively are input to the fully connected layer 230 for processing.

[0124] It should be noted that Figure 3 and Figure 4 The convolutional neural networks shown are merely examples of two possible convolutional neural networks. In specific applications, the convolutional neural networks of the embodiments of the present application may also exist in the form of other network models.

[0125] Figure 5 The hardware structure of a chip provided in an embodiment of the present application includes a neural network processor 50. The chip can be arranged in Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 3 and Figure 4 The algorithms of each layer in the convolutional neural network shown in the figure or the algorithms of each layer in other models in the embodiments of the present application can be as follows Figure 5 is implemented in the chip shown.

[0126] The neural network processor NPU 50 is mounted on the host central processing unit (CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 503, and the controller 504 controls the operation circuit 503 to extract data from the memory (weight memory or input memory) and perform operations.

[0127] In some implementations, the operation circuit 503 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 503 is a two-dimensional systolic array. The operation circuit 503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 503 is a general-purpose matrix processor.

[0128] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 501 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 508.

[0129] The vector calculation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0130] In some implementations, the vector calculation unit 507 can store the processed output vector to the unified buffer 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 503, such as for use in a subsequent layer in a neural network.

[0131] The unified memory 506 is used to store input data and output data.

[0132] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0133] The bus interface unit (BIU) 510 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

[0134] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0135] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0136] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM for short), a high bandwidth memory (HBM) or other readable and writable memory.

[0137] in, Figure 3 and Figure 4 The operations of each layer in the convolutional neural network shown or the operations of each layer in other models in the embodiments of the present application, for example, the operations of each layer in the GAN model, the latent vector encoding model or the image feature extraction model can be performed by the operation circuit 503 or the vector calculation unit 507.

[0138] Introduced above Figure 2 The execution device 110 in the embodiment of the present application can execute each step of the method for generating a text image. Figure 5 The chip shown can also be used to execute the various steps of the method for generating a text image in the embodiment of the present application.

[0139] In the process of training and testing models (e.g., text recognition models), it is often necessary to use text images of a specific style, for example, text images in real scenes. It is difficult to obtain text images of a specific style using existing methods. For example, although the traditional GAN ​​model can generate rich text images, it cannot control the style of the generated text images. Moreover, the GAN model does not have the ability to understand semantics. The text images generated by GAN can only guarantee the shape characteristics of the text as much as possible, but cannot guarantee that the text images have reasonable text semantics. Figure 6 As shown in the figure, the GAN model can only generate text images obtained by randomly splicing single letters / words, and the text in the text image does not have reasonable semantics. For another example, the neural style transfer algorithm can combine the content image and the style image to obtain a fused image after transfer. However, this method can only transfer part of the style in the style image, such as the texture features of the image and the artistic expression of the image. For example, the artistic expression of the image may include cartoons, comics, oil paintings, watercolors, ink paintings and other image styles. This method cannot combine the content image with the image in the real scene, and thus cannot obtain the image in the real scene, such as Figure 7As shown in the figure, the text image generated by this method can only transfer certain texture features, and cannot change other features of the text image, such as the background and font. This will result in a relatively simple generated text image, and it may not be possible to obtain a text image of a specific style, which will affect the subsequent training and testing of the model.

[0140] The method for generating text images proposed in the embodiment of the present application can obtain images of a specific style to meet different user needs, or in other words, meet different model training needs.

[0141] The following describes the method provided by this application from the perspectives of model training and model application:

[0142] The training method of the latent vector coding model provided in the embodiment of the present application involves the processing of computer vision, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning, and the training data (such as the sample data in the present application) is symbolized and formalized for intelligent information modeling, extraction, preprocessing, training, etc., and finally a trained latent vector coding model is obtained; and the method for generating a text image provided in the embodiment of the present application can use the above-mentioned trained latent vector coding model, input the input data (such as the image to be processed in the present application) into the trained latent vector coding model, obtain output data (such as the predicted latent vector in the present application), and then obtain the target image according to the output data. It should be noted that the training method of the latent vector coding model and the method for generating a text image provided in the embodiment of the present application are solutions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as a model training stage and a model application stage.

[0143] The method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0144] Combine the following Figure 8 A training method 900 of a latent vector coding model according to an embodiment of the present application is described in detail. The method 900 can be regarded as a model training stage. The method 900 includes steps S910 to S930. Figure 8 The method shown can be performed by a training device for a latent vector encoding model, which can be a cloud service device or a mobile terminal, such as a computer, a server, an image workstation, etc., which has sufficient computing power to train an image generation model. Figure 2 Training device 120 or Figure 5 The neural network processor 50 in the method 900 is executed, and the sample image set in the method 900 can be Figure 1Optionally, step S910 and step S920 of method 900 may be executed in the training device 120, or may be pre-executed by other functional modules before the training device 120, that is, the training data received or acquired from the database 130 is pre-processed, such as step S910 and step S920, to obtain a latent vector and a text image generated by the latent vector as input to the training device 120, and the training device 120 executes step S930.

[0145] Optionally, the method 900 may be processed by a CPU, or may be processed by a CPU and a GPU together, or may use other processors suitable for neural network calculations without using a GPU, and this application does not impose any limitation thereto.

[0146] Steps S910 to S930 are described in detail below.

[0147] S910, training a GAN model based on the sample image set.

[0148] The sample image set includes a plurality of sample images. The sample image may be a text image. A text image refers to an image containing text.

[0149] The trained GAN model can be used to generate text images.

[0150] Exemplarily, the GAN model may be a StyleGAN model or a StyleGAN2 model, etc.

[0151] S920, using the trained GAN model to sample a first latent vector (Latent) from the latent vector space, and generating a first candidate text image based on the latent vector.

[0152] The sampling of the first latent vector from the latent vector space may be a random sampling of the first latent vector, that is, the first candidate text image may be randomly generated.

[0153] In the embodiment of the present application, the latent vector may also be referred to as a hidden code or a latent vector code or Latent.

[0154] Exemplarily, step S920 may be performed multiple times to obtain multiple first candidate text images. Alternatively, it may be understood that a plurality of first latent vectors are sampled from the latent vector space using the trained GAN model, and multiple first candidate text images are generated based on the multiple first latent vectors. The multiple first candidate text images correspond to the multiple first latent vectors, that is, there is a one-to-one correspondence between the first latent vectors and the first candidate text images.

[0155] Exemplarily, the first candidate text image and the first latent vector corresponding to the first candidate text image may be recorded or saved during the execution of step S920. The first latent vector corresponding to the first candidate text image, or the first latent vector corresponding to the first candidate text image in the GAN model, may be understood as the first candidate text image being able to be generated by the first latent vector in the GAN model.

[0156] S930, using the first candidate text image as an input of a latent vector coding model and using the first latent vector as a target output of the latent vector coding model to train the latent vector coding model.

[0157] It should be noted that step S910 and step S920 are optional steps. The first latent vector and the first candidate text image may also be obtained by other methods. For example, the first latent vector and the first candidate text image may be Figure 1 The training data maintained in the database 130 in . The training data may be data received from other devices. In this case, the GAN model used to generate the first candidate text image according to the first latent vector may also be received from other devices.

[0158] In the embodiment of the present application, the latent vector encoding model may also be referred to as a latent vector encoder.

[0159] The input of the latent vector encoding model is a text image, and the output is a predicted latent vector corresponding to the text image in the GAN model. That is to say, the first candidate text image generated by the GAN model is input into the latent vector encoding model, so that the output predicted latent vector is as close to the first latent vector as possible, or the difference between the output predicted latent vector and the first latent vector is minimized.

[0160] The latent vector encoding model may be a neural network model, that is, the mapping relationship from the text image space to the latent vector space is learned through the neural network model, or the mapping relationship from the first candidate text image to the first latent vector is learned.

[0161] Fig. 9 FIG. 1 shows a schematic block diagram of a GAN model and a latent vector encoding model. Fig. 9 As shown, the latent vector space is Fig. 9 X in the text image space is Fig. 9Y in. Among them, the data in the latent vector space is the latent vector data, and the data in the text image space is the text image data. The GAN model can sample the latent vector x from the latent vector space and generate the candidate text image y. The candidate text image y is input into the latent vector encoding model so that the output prediction latent vector is close to the latent vector x. Among them, the difference between the output prediction latent vector and the latent vector x can be represented by the cycle consistency loss. That is to say, by minimizing the cycle consistency loss, the output prediction latent vector is close to the latent vector x, that is, the latent vector encoding model is trained.

[0162] The trained latent vector encoding model can be used to generate the predicted latent vector corresponding to the text image in the GAN model. For example, the user inputs a real text image into the trained latent vector encoding model and obtains the latent vector encoding corresponding to the real text image. The real image and the image generated by GAN are relative concepts, that is, in the embodiment of the present application, the real image may include an image that is not generated by GAN. The real image may be an image in the sample image set or other images.

[0163] In this way, the latent vector encoder can be used to obtain the predicted latent vector of the real text image in the GAN model, and the characteristics of the text image generated based on the real text image can be controlled by adjusting the predicted latent vector.

[0164] It should be noted that Fig. 9 The values ​​of the latent vectors in are for illustration only and do not constitute a limitation on the solutions of the embodiments of the present application.

[0165] Fig.10 A schematic flow chart of a method 1100 for generating a text image provided by an embodiment of the present application is shown. The method can be executed by a device or apparatus capable of generating an image, for example, the method can be executed by a terminal device, a computer, a server or an image workstation.

[0166] The method 1100 includes steps S1110 to S1140. For example, the method 1100 may be specifically performed as follows: Figure 1 The execution device 110 shown in FIG. 110 is executed, and the text image to be processed in the method 1100 may be as follows: Figure 1The input data provided by the client device 140 shown in the figure, the preprocessing module 113 in the execution device 110 can be used to execute S1120 in the method 1100, the preprocessing module 114 in the execution device 110 can be used to execute S1130 in the method 1100, and the computing module 111 in the execution device 110 can be used to execute S1140. Optionally, the execution device 110 may also include only one preprocessing module, which can be used to execute S1120 and S1130 in the method 1100, and the computing module 111 can be used to execute S1140. Optionally, the execution device 110 may also not include the preprocessing module 113 and the preprocessing module 114, and the computing module 111 directly executes steps S1120 to S1140 in the method 1110.

[0167] Optionally, the method 1100 may be processed by a CPU, or may be processed jointly by a CPU and a GPU, or may use other processors suitable for neural network calculations without using a GPU, and this application does not impose any limitation thereto.

[0168] Fig.10 The GAN model or latent vector encoding model used in the method 1100 for generating a text image can be the above Figure 8 The specific implementation of method 1100 can refer to the aforementioned method 900. In order to avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing method 1100 below.

[0169] S1110, obtaining n text images to be processed, where n≥2 and n is an integer.

[0170] Exemplarily, the text image to be processed may include a content text image and a style text image. Method 1100 may be used to fuse the text content in the content text image with the image style in the style text image to generate a target text image having the text content of the content text image and the image style of the style text image. It should be noted that method 1100 is only described here by taking the example that the text image to be processed includes a content text image and a style text image, and the content text image and the style text image are not divided in an absolute sense. A text image used to provide text content in a target text image can be regarded as a content text image. A text image used to provide an image style in a target text image can be regarded as a style text image. A text image can be both a content text image and a style text image.

[0171] For example, the text image to be processed includes a first text image and a second text image, and method 1100 can be used to fuse feature A in the first text image, feature B in the first text image, feature C in the second text image, and feature D in the second text image to generate a target text image having features A, B, C, and D. Feature A may be a feature indicating the content of the text, for example, feature A may be the shape of the text in the first text image. Feature B, feature C, and feature D may be features indicating the style of the image, for example, feature B may be the color of the text in the first text image, feature C may be the font of the second text image, and feature D may be the background of the second text image. In this case, the second text image may be regarded as a style text image, and the first text image may be regarded as either a style text image or a content text image.

[0172] The content text image may be an image captured by a terminal device (or other devices or equipment such as a computer or server) through a camera, or the content text image may also be an image obtained from inside the terminal device (or other devices or equipment such as a computer or server) (for example, an image stored in an album of the terminal device, or an image obtained by the terminal device from the cloud), and the embodiments of the present application are not limited to this.

[0173] The style text image may be an image captured by a terminal device (or other devices or equipment such as a computer or server) through a camera, or the style text image may also be an image obtained from inside the terminal device (or other devices or equipment such as a computer or server) (for example, an image stored in an album of the terminal device, or an image obtained from the cloud by the terminal device), and the embodiments of the present application are not limited to this.

[0174] The content text image may include one text image or multiple text images, and the style text image may include one text image or multiple text images.

[0175] For ease of description, in the embodiment of the present application, method 900 is described by taking the image to be processed including two text images, namely a first text image and a second text image as an example, and the number of images to be processed in the embodiment of the present application is not limited.

[0176] Fig.11 A schematic flow chart of a method for generating a text image provided by an embodiment of the present application is shown. The first text image may be Fig.11 The content text image in the second text image can be Fig.11 Style text image in .

[0177] S1120, determining n predicted latent vectors corresponding to the n text images to be processed in the GAN model.

[0178] Determining the predicted latent vector corresponding to the text image to be processed in the GAN model can also be understood as embedding the text image to be processed into the latent vector space of the GAN model.

[0179] Optionally, step S1120 may include: inputting n to-be-processed images into a latent vector encoding model for processing, and obtaining n predicted latent vectors corresponding to the n to-be-processed text images in the GAN model.

[0180] The latent vector encoding model can be obtained by training the latent vector encoding model using the first candidate text image as an input of the latent vector encoding model and using the first latent vector corresponding to the first candidate text image in the GAN model as a target output of the latent vector encoding model. The first latent vector is used to generate the first candidate text image in the GAN.

[0181] Specifically, the latent vector encoding model can be trained by method 900, such as Fig.11 As shown, the training process may include the following steps:

[0182] S1: Use the trained GAN model to sample the first latent vector from the latent vector space, and generate the first candidate text image based on the latent vector. For a detailed description, see Figure 8 Method 900 in will not be described in detail here.

[0183] like Fig.11 As shown, the first latent vector includes multiple latent vectors such as X1, X2, ...Xn, etc. The first candidate text image includes multiple candidate text images such as Y1, Y2, ...Yn, etc.

[0184] Exemplarily, the first latent vector and the first candidate text image may constitute data pairs (Y1, X1), (Y2, X2)…(Yn, Xn), etc., for training a latent vector encoding model.

[0185] S2: Train a latent vector encoding model based on the first latent vector and the first candidate text image. For a detailed description, see Figure 8 Method 900 in will not be described in detail here.

[0186] The first text image and the second text image are input into the trained latent vector encoding model to obtain a predicted latent vector corresponding to the first text image and a predicted latent vector corresponding to the second text image.

[0187] like Fig.11 As shown, the content text image Y' is input into the trained latent vector encoding model to obtain the predicted latent vector X' corresponding to Y' in the GAN model, that is, Fig.11 S3 in. Fig.11In the figure, only the content text image is used as an example. The style text image can be processed in the same way to obtain the predicted latent vector corresponding to the style text image.

[0188] Method 900 can be understood as the training phase of a latent vector encoding model (e.g. Figure 1 The specific training is performed by the neural network provided in method 900; and method 1100 can be understood as the application stage of the latent vector encoding model (such as Figure 1 The execution stage executed by the execution device 110 shown in the figure) can be specifically embodied as adopting the latent vector encoding model trained by the method 900, and obtaining the predicted latent vector corresponding to the text image to be processed according to the text image to be processed input by the user.

[0189] It should be noted that step S1120 may not adopt the latent vector encoding model, but may adopt other methods to determine the predicted latent vector of the text image to be processed in the GAN model, such as method 1300 described later.

[0190] S1130, determining a target latent vector according to the n predicted latent vectors.

[0191] The target latent vector encoding can be used to generate the target image.

[0192] Specifically, the n predicted latent vectors can be processed according to the control mode of the latent vector on the generated text image in the GAN model to obtain the target latent vector.

[0193] Among them, the way in which the latent vector controls the generated text image in the GAN model refers to the influence of the elements in the latent vector on the image features in the generated text image. The influence of the elements in the latent vector on the image features in the generated text image can also be understood as the relationship between the elements in the latent vector and the image features.

[0194] Different elements in a latent vector encoding can control different features of the generated image. For example, one element in a latent vector encoding can control a feature of the generated text image. Alternatively, multiple elements in a latent vector encoding can jointly control a feature of the generated text image. In other words, the feature of the generated text image can be controlled by one element in a latent vector encoding, or by multiple elements.

[0195] Optionally, method S1130 further includes: determining the relationship between the elements in the latent vector and the image features. This step corresponds to Fig.11 Step S4 in .

[0196] Exemplarily, by changing the values ​​of the elements in the latent vector and comparing the differences between the text images generated by the latent vectors before and after the change, it can be determined that different elements in the latent vector correspond to different image features of the generated text image in the GAN model. For example, element P can be used to control feature P of the generated text image. Element Q can be used to control feature Q of the generated text image. Element P can be understood as the element at position P in the latent vector. Element Q can be understood as the element at position Q in the latent vector.

[0197] The relationship between the elements in the latent vector and the image features may be determined in the above manner or in other manners. For example, the relationship may be pre-stored.

[0198] Exemplarily, image features may include content features and style features. For example, content features may include text shape. Style features may include text color, background, text font, or light, etc.

[0199] In the embodiment of the present application, image features may also be referred to as factors.

[0200] Optionally, step S1130 may include: cross-mixing the n prediction latent vectors to obtain a target latent vector, wherein the target latent vector includes at least one element in each of the n prediction latent vectors, and at least one element in each of the prediction latent vectors is used to control a target image feature of the target text image.

[0201] Specifically, the target element for controlling the target image feature can be determined according to the relationship between the elements in the latent vector and the image feature, and then the n predicted latent vectors can be cross-mixed according to the target element.

[0202] For example, the target image feature includes the feature P of the first text image and the feature Q of the second text image, and the target element is determined according to the relationship between the element in the latent vector and the image feature. The target element includes the element P in the prediction latent vector corresponding to the first text image and the element Q in the prediction latent vector corresponding to the second text image, and the element Q in the prediction latent vector corresponding to the first text image can be replaced with the element Q in the prediction latent vector corresponding to the second text image, and the corresponding prediction latent vector of the replaced first text image is used as the target latent vector.

[0203] It should be noted that the element P may be one element or multiple elements. The element Q may be one element or multiple elements. The feature P may be one image feature or multiple image features. The feature Q may be one image feature or multiple image features.

[0204] like Fig.11As shown, the target image features include the text shape of the content text image and the text font and background of the style text image. Fig.11 Step S5 in can correspond to step S1130. Specifically, the target latent vector can be determined according to the elements controlling the text shape in the prediction latent vector [63, 7, 92, ... 458] corresponding to the content text image and the elements controlling the text font and the elements controlling the background in the prediction latent vector [88, 34, 22, ... 325] corresponding to the style text image. For example, the elements controlling the text font and the elements controlling the background in the prediction latent vector corresponding to the content text image are used to replace the elements controlling the text shape in the prediction latent vector corresponding to the content text image, that is, the numerical values ​​at the positions controlling the text font and the numerical values ​​at the positions controlling the background in the prediction latent vector corresponding to the style text image are used to replace the data at the positions controlling the text shape in the prediction latent vector corresponding to the content text image, and the prediction latent vector corresponding to the replaced content text image is used as the target latent vector.

[0205] It should be noted that Fig.11 The values ​​of the latent vectors and the fusion method are for illustration only and do not constitute a limitation on the solutions of the embodiments of the present application.

[0206] S1140, generating a target text image by encoding the target latent vector through a GAN model.

[0207] For example, Fig.11 In step S6, the generated target text image includes the text shape in the content text image and the background and text font in the style text image.

[0208] According to the scheme of the embodiment of the present application, the target latent vector is determined by the predicted latent vector corresponding to the text image to be processed, and then the characteristics of the target text image can be controlled by the target latent vector. Different characteristics of the image determine the different styles displayed by the image. Through the scheme of the embodiment of the present application, the target latent vector can be flexibly adjusted according to the required style to generate a text image with a specific style to meet the application requirements.

[0209] In addition, since the GAN model does not have semantic perception capabilities, the text in the text image generated directly by the GAN model may not have reasonable semantics. In the solution of the embodiment of the present application, the semantic processing process can be bypassed, and the predicted latent vector corresponding to the text image to be processed can be determined, so that the text image to be processed can retain the correct semantics of the text, or in other words, the correct semantics in the text image to be processed can be retained in the target text image.

[0210] For example, the text image to be processed may include a content text image and a style text image. The predicted latent vector corresponding to the content text image can generate a text image close to the content text image, and the shape of the text in the content text image can be retained, which will not affect the semantics of the text in the image. In this way, when the GAN model is unaware of semantics, by retaining the elements that control the shape of the text in the content text image in the target latent vector, the target latent vector can be used to generate a target text image with the shape of the text in the content text image, that is, a target text image with reasonable semantics.

[0211] In addition, according to the scheme of the embodiment of the present application, the required elements in the target latent vector can be determined according to the required image features in the target text image, or in other words, the mixing method of the predicted latent vector can be determined to obtain the target latent vector that can generate the target text image, which can further improve the controllability of the style of the generated target text image and obtain the desired target text image.

[0212] In addition, according to the scheme of the embodiment of the present application, it is possible to change the text shape, background or font and other features of the text image to be processed to generate a target text image. Compared with the scheme that cannot change specific features such as the text font, this scheme greatly improves the diversity of the generated target text images.

[0213] It should be understood that the embodiment of the present application only takes the generation of a target text image based on two to-be-processed text images as an example, and does not limit the method for generating a text image provided by the embodiment of the present application. The method for generating a text image provided by the embodiment of the present application can also be used to generate a target text image based on more than two to-be-processed text images.

[0214] The following takes the example of generating a target text image based on three to-be-processed text images to illustrate the method of generating a text image in the embodiment of the present application. The detailed process can be found in the description of method 1100. To avoid repetition, the following description is appropriately omitted.

[0215] (1) Obtaining text images to be processed, where the text images to be processed include a first text image, a second text image, and a third text image.

[0216] (2) Determine a prediction latent vector corresponding to the first text image, a prediction latent vector corresponding to the second text image, and a prediction latent vector corresponding to the third text image.

[0217] (3) Obtaining a target latent vector based on the predicted latent vector. The target latent vector includes a first element in the predicted latent vector corresponding to the first text image, a second element in the predicted latent vector corresponding to the second text image, and a third element in the predicted latent vector corresponding to the third text image. The target image feature includes a first feature, a second feature, and a third feature. The first element includes an element for controlling a first text feature of the target text image, the second element includes an element for controlling a second feature of the target text image, and the third element includes an element for controlling the generation of a third feature of the target text image.

[0218] For example, the target image features include the text shape in the first text image, the text font in the second text image, and the background in the third text image. The first feature includes the text shape of the first text image, and the first element is used to control the text shape in the target text image to be the text shape of the first text image. The second feature includes the text font in the second text image, and the second element is used to control the text font in the target text image to be the text font in the second text image. The third feature includes the background in the third text image, and the third element is used to control the background in the target text image to be the background in the third text image. The target text image generated in this way includes the text shape in the first text image, the text font in the second text image, and the background in the third text image.

[0219] (4) Generate the target text image by encoding the target latent vector through the GAN model.

[0220] Fig.12 A method 1300 for determining a latent vector encoding corresponding to an image to be processed provided by an embodiment of the present application is shown. The method 1300 can be regarded as a specific implementation of step S1120. The method 1300 includes steps S1310 to S1340, and steps S1310 to S1340 are described in detail below.

[0221] S1310, training a GAN model based on the sample image set.

[0222] The sample image set includes a plurality of sample images. The sample images may be text images.

[0223] It should be noted that step S1310 is an optional step, and method 1300 can start directly from step S1320. Exemplarily, the trained GAN model can be a pre-trained GAN model, or it can also be a GAN model received from other devices, or a GAN model input by a user.

[0224] S1320, sample a second latent vector using the trained GAN model.

[0225] Specifically, a second latent vector is sampled from the latent vector space using a trained GAN model.

[0226] The sampling of the second latent vector from the latent vector space may be random sampling of the second latent vector.

[0227] S1330: Generate a second candidate text image according to the second latent vector.

[0228] S1340: Determine the difference between the second candidate text image and the first text image.

[0229] The first text image is one of the n text images to be processed.

[0230] Exemplarily, the difference between two text images can be determined by feature distance. The smaller the feature distance between two text images, the smaller the difference between the two image texts. The feature distance between two text images refers to the difference between the feature values ​​of the two text images.

[0231] Optionally, step S1340 includes:

[0232] Performing feature extraction on the second candidate text image to obtain a feature value of the second candidate text image;

[0233] Performing feature extraction on the first text image to obtain a feature value of the first text image;

[0234] The difference between the second candidate text image and the first text image is determined based on the difference between the feature value of the second candidate text image and the feature value of the first text image. For example, the difference between the feature value of the second candidate text image and the feature value of the first text image is used as the difference between the second candidate text image and the first text image.

[0235] Specifically, the image feature extraction model can be used to extract features from the image to obtain the feature value of the image. For example, the image feature extraction model can be a visual geometry group network (VGG-16) network model.

[0236] S1350: Update the second latent vector, and use the updated second latent vector as the second latent vector in step S1330.

[0237] Repeat steps S1330 to S1350 until the termination condition is met, and use the updated second latent vector as the predicted latent vector corresponding to the first text image in the GAN model.

[0238] Among them, the termination condition includes at least one of the following: the difference between the second candidate text image and the first text image is less than or equal to the first threshold, or the number of repetitions is greater than or equal to the second threshold. It should be noted that the termination condition can be set as needed, and the embodiment of the present application does not limit the termination condition. For example, the termination condition can also be that the difference between the second candidate text image generated by the second latent vector after this update and the second candidate text image generated by the second latent vector after the last update is less than or equal to the third threshold. In other words, the iteration is terminated when the difference between the generated second candidate text image and the first text image changes slightly.

[0239] That is, method 1300 determines the prediction latent vector corresponding to the text image to be processed by minimizing the difference between the text image to be processed and the second candidate text image.

[0240] It should be noted that determining the prediction latent vector corresponding to the text image to be processed by minimizing the difference between the text image to be processed and the second candidate text image may also include other implementation forms.

[0241] For example, the prediction latent vector can also be determined by the following steps.

[0242] Step 1: Use the trained GAN model to sample multiple second latent vectors.

[0243] Step 2: Generate multiple second candidate text images according to the multiple second latent vectors.

[0244] Step three: taking the latent vector corresponding to the second candidate text image with the smallest difference from the first text image among the plurality of second candidate text images as the predicted latent vector of the first text image.

[0245] In method 1300 , only the first text image is taken as an example, and other text images among the n text images to be processed can perform the same operation as the first text image to obtain the prediction latent vectors corresponding to the n text images to be processed.

[0246] Fig.13 A method for generating a text image is shown. This method can be regarded as a specific implementation of method 1300 and method 1100. For a detailed description, please refer to the aforementioned method 1100 and method 1300 and Fig.11 In order to avoid unnecessary repetition, Fig.13 Make appropriate omissions when using the method.

[0247] S1, using an image feature extraction model to extract features of the text image to be processed, and obtaining feature values ​​of the text image to be processed.

[0248] For example, Fig.13As shown, the text to be processed includes a content text image and a style text image. The content text image Y' is input into the image feature extraction model to obtain a feature value Z' of the content text image.

[0249] It should be noted that Fig.13 The feature value extraction of the content text image is only used as an example. For the style text image, the same operation as that of the content text image can be performed, that is, the style text image is input into the image feature extraction model to obtain the feature value of the style text image.

[0250] S2, using the GAN model to sample the second latent vector, and generating a second candidate text image based on the second latent vector; using the image feature extraction model to extract features of the second candidate text image, and obtaining feature values ​​of the second candidate text image.

[0251] For example, Fig.13 As shown, a second candidate text image Y is generated based on the second latent vector X, and the second candidate text image Y is input into the image feature extraction model to obtain a feature value Z of the second candidate text image.

[0252] S3, determining a prediction latent vector corresponding to the text image to be processed by minimizing the feature distance between the text image to be processed and the second candidate text image.

[0253] Steps S4 to S6 and Fig.11 The steps are the same as in , and will not be repeated here.

[0254] It should be noted that Fig.13 The values ​​of the latent vectors and the fusion method are for illustration only and do not constitute a limitation on the solutions of the embodiments of the present application.

[0255] Fig.14 It is a schematic diagram of the hardware structure of the device for generating text images according to an embodiment of the present application. Fig.14 The device 4000 for generating text images shown includes a memory 4001, a processor 4002, a communication interface 4003 and a bus 4004. The memory 4001, the processor 4002 and the communication interface 4003 are connected to each other through the bus 4004.

[0256] The memory 4001 may be a ROM, a static storage device, or a RAM. The memory 4001 may store a program. When the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute the various steps of the method for generating a text image in the embodiment of the present application. Specifically, the processor 4002 may execute the above Fig.10 Optionally, the processor 4002 may also execute the steps S1110 to S1140 in the method shown above. Fig.11 or Fig.13 Optionally, the processor 4002 may also execute the steps S1 to S6 in the method shown above. Fig.12 Steps S1310 to S1340 in the method shown.

[0257] Processor 4002 can be a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits, which are used to execute relevant programs to implement the functions required to be performed by the units in the device for generating text images in the embodiment of the present application, or to execute the method for generating text images in the method embodiment of the present application.

[0258] The processor 4002 may also be an integrated circuit chip having the ability to process signals, for example, Figure 5 In the implementation process, each step of the method for generating a text image in the embodiment of the present application can be completed by an integrated logic circuit of hardware in the processor 4002 or by instructions in the form of software.

[0259] Processor 4002 is used to: obtain n text images to be processed, where n≥2 and n is an integer; determine n prediction latent vectors corresponding to the n text images to be processed in a generative adversarial network (GAN) model; determine a target latent vector based on the n prediction latent vectors; and generate a target text image from the target latent vector through the GAN model.

[0260] Optionally, determining the target latent vector based on the n prediction latent vectors includes: cross-mixing the n prediction latent vectors to obtain the target latent vector, wherein the target latent vector includes at least one element in each of the n prediction latent vectors, and at least one element in each of the prediction latent vectors is used to control the target image features of the target text image.

[0261] Optionally, the target image features include the text shape, background or text font.

[0262] Optionally, determining the n predicted latent vectors corresponding to the n text images to be processed in the GAN model includes: inputting the n images to be processed into a latent vector encoding model for processing to obtain n predicted latent vectors corresponding to the n text images to be processed in the GAN model; wherein the latent vector encoding model is obtained by training the latent vector encoding model with a first candidate text image as the input of the latent vector encoding model and with a first latent vector corresponding to the first candidate text image in the GAN model as the target output of the latent vector encoding model, and the first latent vector is used to generate the first candidate text image in the GAN.

[0263] Optionally, determining the n predicted latent vectors corresponding to the n text images to be processed in the GAN model includes: step one: sampling a second latent vector using the GAN model; step two: generating a second candidate text image based on the second latent vector; step three: determining the difference between the second candidate text image and the first text image, the first text image being one of the n text images to be processed; step four: updating the second latent vector, and using the updated second latent vector as the second latent vector in step two; repeating steps two to four until a termination condition is met, and using the updated second latent vector as the predicted latent vector corresponding to the first text image in the GAN model, wherein the termination condition includes at least one of the following: the difference between the second candidate text image and the first text image is less than or equal to a first threshold, or the number of repetitions is greater than or equal to a second threshold.

[0264] Optionally, the determining the difference between the second candidate text image and the first text image includes: performing feature extraction on the second candidate text image to obtain a feature value of the second candidate text image;

[0265] Perform feature extraction on the first text image to obtain a feature value of the first text image; and use the difference between the feature value of the second candidate text image and the feature value of the first text image as the difference between the second candidate text image and the first text image.

[0266] The processor 4002 may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 4001, and the processor 4002 reads the information in the memory 4001, and completes the functions required to be executed by the units included in the device for generating a text image in the embodiments of the present application in combination with its hardware, or executes the method for generating a text image in the embodiments of the present application.

[0267] The communication interface 4003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 4000 and other devices or a communication network. For example, the text image to be processed can be obtained through the communication interface 4003.

[0268] The bus 4004 may include a path for transmitting information between various components of the device 4000 (eg, the memory 4001 , the processor 4002 , and the communication interface 4003 ).

[0269] Fig.15 4000 is a schematic diagram of the hardware structure of the training device of the embodiment of the present application. Fig.15 The training device 5000 shown includes a memory 5001, a processor 5002, a communication interface 5003 and a bus 5004. The memory 5001, the processor 5002 and the communication interface 5003 are connected to each other through the bus 5004.

[0270] pass Fig.15 The training device 5000 shown trains the GAN model or the latent vector encoding model, and the trained GAN model or the latent vector encoding model can be used to execute the method for generating a text image according to an embodiment of the present application.

[0271] Specifically, Fig.15 The device shown can obtain training data and the GAN model or latent vector encoding model to be trained from the outside world through the communication interface 5003, and then the processor trains the GAN model or latent vector encoding model to be trained according to the training data.

[0272] The memory 5001 may be a ROM, a static storage device, or a RAM. The memory 5001 may store a program. When the program stored in the memory 5001 is executed by the processor 5002, the processor 4002 and the communication interface 5003 are used to execute the various steps of the training method of the latent vector coding model of the embodiment of the present application. Specifically, the processor 5002 may execute the above Figure 8 Steps S910 to S930 in the method shown.

[0273] The processor 5002 can be a general-purpose CPU, a microprocessor, an ASIC, a GPU or one or more integrated circuits, which are used to execute relevant programs to implement the functions required to be performed by the units in the training device of the embodiment of the present application, or to execute the training method of the latent vector coding model of the method embodiment of the present application.

[0274] The processor 5002 may also be an integrated circuit chip having the ability to process signals, for example, Figure 5In the implementation process, each step of the training method of the latent vector coding model in the embodiment of the present application can be completed by the hardware integrated logic circuit in the processor 5002 or the instruction in the form of software.

[0275] The processor 5002 may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 5001, and the processor 5002 reads the information in the memory 5001, and completes the functions required to be performed by the unit in the training device of the embodiment of the present application in combination with its hardware, or executes the training method of the latent vector coding model of the method embodiment of the present application.

[0276] The communication interface 5003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 5000 and other devices or a communication network. For example, a sample image set can be acquired through the communication interface 5003 .

[0277] The bus 5004 may include a path for transmitting information between various components of the device 5000 (eg, the memory 5001 , the processor 4002 , and the communication interface 5003 ).

[0278] It should be noted that although the above-mentioned apparatus 4000 and apparatus 5000 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the apparatus 4000 and apparatus 5000 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the apparatus 4000 and apparatus 5000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the apparatus 4000 and apparatus 5000 may also only include the devices necessary for implementing the embodiments of the present application, and do not necessarily include Fig.14 and Fig.15 All devices shown in .

[0279] It should be understood that the processor in the embodiment of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0280] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0281] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0282] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0283] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0284] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0285] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0286] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0287] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0288] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0289] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0290] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0291] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating a text image, characterized in that: include: Obtain n text images to be processed, where n is an integer greater than or equal to 2; Determine n predicted latent vectors corresponding to the n text images to be processed in a generative adversarial network (GAN) model; Determine a target latent vector according to the n predicted latent vectors; A target text image is generated from the target latent vector through the GAN model.

2. The method according to claim 1, characterized in that The determining a target latent vector according to the n predicted latent vectors includes: The n prediction latent vectors are cross-mixed to obtain the target latent vector, wherein the target latent vector includes at least one element in each of the n prediction latent vectors, and at least one element in each of the prediction latent vectors is used to control the target image feature of the target text image.

3. The method according to claim 2, characterized in that The target image features include text shape, background or text font.

4. The method according to any one of claims 1 to 3, characterized in that The determining of n predicted latent vectors corresponding to the n to-be-processed text images in the GAN model includes: Inputting the n to-be-processed text images into a latent vector encoding model for processing, and obtaining n predicted latent vectors corresponding to the n to-be-processed text images in a GAN model; Among them, the latent vector encoding model is obtained by training the latent vector encoding model with a first candidate text image as the input of the latent vector encoding model and a first latent vector corresponding to the first candidate text image in the GAN model as the target output of the latent vector encoding model, and the first latent vector is used to generate the first candidate text image in the GAN.

5. The method according to any one of claims 1 to 3, characterized in that The determining of n predicted latent vectors corresponding to the n to-be-processed text images in the GAN model includes: Step 1: Sample a second latent vector using the GAN model; Step 2: generating a second candidate text image according to the second latent vector; Step 3: determining the difference between the second candidate text image and the first text image, where the first text image is one of the n text images to be processed; Step 4: Update the second latent vector, and use the updated second latent vector as the second latent vector in step 2; Repeat step 2 to step 4 until the termination condition is met, and use the updated second latent vector as the predicted latent vector corresponding to the first text image in the GAN model. The termination condition includes at least one of the following: the difference between the second candidate text image and the first text image is less than or equal to a first threshold, or the number of repetitions is greater than or equal to a second threshold.

6. The method according to claim 5, characterized in that The determining the difference between the second candidate text image and the first text image includes: Performing feature extraction on the second candidate text image to obtain a feature value of the second candidate text image; Performing feature extraction on the first text image to obtain a feature value of the first text image; The difference between the feature value of the second candidate text image and the feature value of the first text image is used as the difference between the second candidate text image and the first text image.

7. A device for generating a text image, characterized in that: include: An acquisition unit, used for acquiring n text images to be processed, where n is an integer greater than or equal to 2; A processing unit for: Determine n predicted latent vectors corresponding to the n text images to be processed in a generative adversarial network (GAN) model; Determine a target latent vector according to the n predicted latent vectors; A target text image is generated from the target latent vector through the GAN model.

8. The device according to claim 7, characterized in that The processing unit is specifically used for: The n prediction latent vectors are cross-mixed to obtain the target latent vector, wherein the target latent vector includes at least one element in each of the n prediction latent vectors, and at least one element in each of the prediction latent vectors is used to control the target image feature of the target text image.

9. The device according to claim 8, characterized in that The target image features include text shape, background or text font.

10. The device according to any one of claims 7 to 9, characterized in that The processing unit is specifically used for: Inputting the n to-be-processed text images into a latent vector encoding model for processing, and obtaining n predicted latent vectors corresponding to the n to-be-processed text images in a GAN model; Among them, the latent vector encoding model is obtained by training the latent vector encoding model with a first candidate text image as the input of the latent vector encoding model and a first latent vector corresponding to the first candidate text image in the GAN model as the target output of the latent vector encoding model, and the first latent vector is used to generate the first candidate text image in the GAN.

11. The device according to any one of claims 7 to 9, characterized in that The processing unit is specifically used for: Step 1: Sample a second latent vector using the GAN model; Step 2: generating a second candidate text image according to the second latent vector; Step 3: determining the difference between the second candidate text image and the first text image, where the first text image is one of the n text images to be processed; Step 4: Update the second latent vector, and use the updated second latent vector as the second latent vector in step 2; Repeat step 2 to step 4 until the termination condition is met, and use the updated second latent vector as the predicted latent vector corresponding to the first text image in the GAN model. The termination condition includes at least one of the following: the difference between the second candidate text image and the first text image is less than or equal to a first threshold, or the number of repetitions is greater than or equal to a second threshold.

12. The device according to claim 11, characterized in that The processing unit is specifically used for: Performing feature extraction on the second candidate text image to obtain a feature value of the second candidate text image; Performing feature extraction on the first text image to obtain a feature value of the first text image; The difference between the feature value of the second candidate text image and the feature value of the first text image is used as the difference between the second candidate text image and the first text image.

13. A device for generating a text image, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The computer-readable medium stores a program code for execution by a device, the program code including a program for executing the method according to any one of claims 1 to 6.

15. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method according to any one of claims 1 to 6.

Images