Image text processing method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of image text processing, and provides an image text processing method and device, electronic equipment and a medium. The method comprises the following steps: when prompt text data contains hand feature information, constructing a hand feature vector according to a feature vector corresponding to the prompt text data; determining an initial image according to the feature vector corresponding to the prompt text data and the feature vector corresponding to the to-be-recognized image; detecting the initial image to determine whether the initial image contains a hand region; when the initial image contains the hand region, determining a feature vector corresponding to the hand region; carrying out fusion processing on the three vectors to obtain a hand fusion feature vector; and generating a target image based on the initial image, the feature vector corresponding to the hand region, the hand fusion feature vector and the feature vector corresponding to the prompt text data. According to the method, the defects of an existing model in hand image generation are overcome, and different types of prompt text and image input can be flexibly adapted.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image text processing, and in particular to an image text processing method, device, electronic device and medium. Background Art

[0002] With the rapid development of Internet technology, text-to-image generation models have made significant progress in recent years. Models such as Stable Diffusion, Imagen, and GLIDE have been able to generate high-quality and realistic images, which are widely used in art creation, advertising design, and virtual reality. However, although these models perform well in generating human images, they still face many challenges in generating realistic hand images. Specifically, common problems in generating hand images include: unrealistic hand postures, irregular hand shapes, incorrect number of fingers, and unnatural interaction between hands and objects. These problems seriously affect the realism and usability of generated images, and limit the effectiveness of text-to-image generation technology in certain application scenarios. In order to solve these problems, some research works in the existing technical solutions have attempted to improve the generation of hands through key point skeletons and shape formats. These methods have shown their effectiveness in the generation tasks of pose animation and hand-object interaction, and provide a basic understanding of reasonable hand shapes. However, how to learn these representations from text cues and effectively map these representations to the pixel space of the image remains an unsolved problem. Therefore, in response to the above challenges, the present disclosure proposes a new image text processing method. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide an image text processing method, device, electronic device and computer-readable storage medium to solve the technical problem in the prior art of how to learn these representations from text prompts and effectively map these representations to the pixel space of the image.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for image text processing is provided, comprising: obtaining prompt text data and an image to be identified; when the prompt text data contains hand feature information, constructing a hand feature vector according to a feature vector corresponding to the prompt text data; determining an initial image according to a feature vector corresponding to the prompt text data and a feature vector corresponding to the image to be identified; detecting the initial image to determine whether the initial image contains a hand area; when the initial image contains a hand area, determining a feature vector corresponding to the hand area; fusing the feature vector corresponding to the prompt text data, the hand feature vector, and the feature vector corresponding to the hand area to obtain a hand fusion feature vector; and generating a target image based on the initial image, the feature vector corresponding to the hand area, the hand fusion feature vector, and the feature vector corresponding to the prompt text data.

[0005] According to a second aspect of an embodiment of the present disclosure, there is provided an image text processing device, comprising: an acquisition module for acquiring prompt text data and an image to be identified; a hand feature construction module for constructing a hand feature vector according to a feature vector corresponding to the prompt text data when the prompt text data contains hand feature information; a first determination module for determining an initial image according to a feature vector corresponding to the prompt text data and a feature vector corresponding to the image to be identified; a second determination module for detecting the initial image to determine whether the initial image contains a hand area; a third determination module for determining a feature vector corresponding to the hand area when the initial image contains a hand area; a fusion module for fusing the feature vector corresponding to the prompt text data, the hand feature vector, and the feature vector corresponding to the hand area to obtain a hand fusion feature vector; and a generation module for generating a target image based on the initial image, the feature vector corresponding to the hand area, the hand fusion feature vector, and the feature vector corresponding to the prompt text data.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0008] Compared with the prior art, the disclosed embodiment has the following beneficial effects: the disclosed embodiment can effectively identify and extract hand features by acquiring prompt text data and the image to be identified and analyzing the hand feature information therein, thereby ensuring that the generated image better meets the user's expectations. By constructing a hand feature vector and fusing it with the prompt text data and the feature vector of the image to be identified, it is possible to integrate a variety of information and improve the accuracy and naturalness of hand image generation. An initial image detection step is introduced during the generation process to confirm the existence of the hand area. This process can ensure the rationality of the hand area in the generated target image and avoid generating unnatural or unrealistic hand postures. By fusing the hand area feature vector with other related feature vectors, the generated target image will be more realistic in terms of hand shape, posture and its interaction with objects, thereby significantly improving the overall quality of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied;

[0011] Figure 2 It is a flowchart of an image text processing method provided by an embodiment of the present disclosure;

[0012] Figure 3 is a schematic diagram of an application scenario of the image text processing method provided by an embodiment of the present disclosure;

[0013] Figure 4 is a block diagram of an image text processing device provided by an embodiment of the present disclosure;

[0014] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0016] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0017] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied is shown.

[0018] like Figure 1As shown, the system architecture 100 may include one or more of a first terminal device 101, a second terminal device 102, and a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0019] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0020] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send data, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, and the like.

[0021] The server 105 may be a server that provides various services. For example, the server 105 may obtain the prompt text data and the image to be identified from the first terminal device 103 (or the second terminal device 102 or the third terminal device 103); when the prompt text data contains hand feature information, a hand feature vector is constructed according to the feature vector corresponding to the prompt text data; an initial image is determined according to the feature vector corresponding to the prompt text data and the feature vector corresponding to the image to be identified; the initial image is detected to determine whether the initial image contains a hand area; when the initial image contains a hand area, a feature vector corresponding to the hand area is determined; the feature vector corresponding to the prompt text data, the hand feature vector, and the feature vector corresponding to the hand area are fused to obtain a hand fusion feature vector; a target image is generated based on the initial image, the feature vector corresponding to the hand area, the hand fusion feature vector, and the feature vector corresponding to the prompt text data.

[0022] In some embodiments, the image text processing method provided by the embodiment of the present invention is generally executed by the server 105, and accordingly, the image text processing device is generally set in the server 105. In other embodiments, some terminal devices may have functions similar to those of the server to execute the method. Therefore, the image text processing method provided by the embodiment of the present invention is not limited to being executed on the server side.

[0023] The image text processing method and device according to the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0024] Figure 2 is a flowchart of an image text processing method provided by an embodiment of the present disclosure. The method provided by the embodiment of the present disclosure can be executed by any electronic device with computer processing capabilities, for example, the electronic device can be Figure 1 The server is shown.

[0025] like Figure 2 As shown, the image text processing method includes steps S210 to S270.

[0026] In step S210, the prompt text data and the image to be recognized are obtained.

[0027] Step S220: When the prompt text data contains hand feature information, a hand feature vector is constructed according to a feature vector corresponding to the prompt text data.

[0028] In step S230, an initial image is determined according to a feature vector corresponding to the prompt text data and a feature vector corresponding to the image to be recognized.

[0029] In step S240, the initial image is detected to determine whether the initial image contains a hand area.

[0030] In step S250, when the initial image includes a hand region, a feature vector corresponding to the hand region is determined.

[0031] In step S260, the feature vector corresponding to the prompt text data, the hand feature vector, and the feature vector corresponding to the hand area are fused to obtain a hand fused feature vector.

[0032] In step S270, a target image is generated based on the initial image, the feature vector corresponding to the hand area, the hand fusion feature vector, and the feature vector corresponding to the prompt text data.

[0033] This method can effectively identify and extract hand features by acquiring prompt text data and the image to be identified and analyzing the hand feature information therein, ensuring that the generated image is more in line with the user's expectations. By constructing a hand feature vector and fusing it with the feature vectors of the prompt text data and the image to be identified, it is possible to integrate a variety of information and improve the accuracy and naturalness of hand image generation. An initial image detection step is introduced in the generation process to confirm the existence of the hand area. This process can ensure the rationality of the hand area in the generated target image and avoid generating unnatural or unrealistic hand postures. By fusing the hand area feature vector with other related feature vectors, the generated target image will be more realistic in terms of hand shape, posture and its interaction with objects, thereby significantly improving the overall quality of the image.

[0034] In some embodiments of the present disclosure, the above-mentioned prompt text data may be text input by the user through an application of a terminal device. This method allows the user to directly describe the hand features according to their needs, for example: "Please generate a right hand holding a cup" or "Generate a hand with fingers pointing forward". This flexibility enables users to accurately express their expectations, thereby improving the relevance and accuracy of the generated results. In addition to the text entered by the user in real time, the background server of the application can also store historical texts. These historical texts are the user's past input or descriptions of other users, and the application can automatically recommend relevant prompt texts based on the context and user preferences. This method not only improves the user experience, but also improves the generation quality by using existing data. The image to be identified can be a reference image uploaded by the user to the application. These images are photos taken by the user or images obtained from other sources, containing feature information of the hand. This process provides a specific visual reference for generating the target image, so that the generated hand image is closer to the actual needs. The image to be identified can also be obtained from the database of the application. These images are pre-stored samples or examples, covering a variety of hand postures and shapes. By selecting suitable images from the database, users can quickly find reference images that meet their needs, further improving the generation efficiency. The prompt text data may contain information for describing hand features. This information includes but is not limited to the hand posture, shape, number of fingers, hand movements (such as gripping, pointing, placing, etc.), and the interaction between the hand and other objects. Through detailed descriptions, the model can better understand the user's intentions and generate hand images that are more in line with expectations. Recognition images can also contain hand images. These images provide an important visual basis for generating target images, making feature extraction and fusion in the generation process more effective. The presence of hand images helps the model capture detailed features of the hand, such as the shape of the fingertips, the outline of the palm, and the bending state of the fingers, thereby improving the realism and naturalness of the final generated image. The above method can flexibly obtain prompt text data and images to be recognized, which can effectively improve the accuracy and naturalness of hand image generation and provide users with a better generation experience.

[0035] In some embodiments of the present disclosure, the above method can also determine whether the above prompt text data contains hand feature information. For example, in order to determine whether the prompt text data contains hand feature information, natural language processing (NLP) technology can be used for text analysis. Specifically, a keyword extraction algorithm (such as TF-IDF, TextRank, etc.) is used to identify important words in the text to determine whether there are keywords related to the hand. The named entity recognition (NER) method is applied to identify whether entities such as "hand", "finger", and "palm" are mentioned in the text. Through semantic analysis methods (such as Word2Vec, BERT, etc.), the overall meaning of the text is understood to determine whether it is related to hand features. The pattern matching method is used to determine whether there is hand feature information in the text through a predefined pattern or template. For example, a set of regular expressions or templates can be created to match sentence structures containing keywords such as "hand", "grip", and "finger". In this way, the judgment is combined with context information, and the recognition of hand feature information not only depends on a single word, but also needs to consider the overall structure and context of the sentence. For example, in the sentence "Please generate a hand holding a mobile phone", although "mobile phone" is not a hand feature, the presence of hand features can be inferred by combining it with "holding". In some cases, a user feedback mechanism can be configured to allow users to confirm or correct the system's recognition results of hand feature information. This method can continuously optimize the model's judgment ability and improve the accuracy of subsequent generation. When it is determined that the prompt text data contains hand feature information, subsequent steps will be continued, such as constructing a hand feature vector and generating a target image.

[0036] In some embodiments of the present disclosure, before constructing the hand feature vector according to the feature vector corresponding to the prompt text data, the method further includes: encoding the prompt text data through a text encoder to obtain the feature vector corresponding to the prompt text data; encoding the image to be recognized through an image encoder to obtain the feature vector corresponding to the image to be recognized. For example, a CLIP text encoder is used. CLIP (Contrastive Language-Image Pre-training) is a model trained by a contrastive learning method, which can map text and images into the same feature space. The prompt text data (for example, a sentence describing a gesture or action) is input into the text encoder. The text encoder converts the input text into a feature vector of fixed dimension. Specifically, the text is segmented, stop words are removed, and padding is performed to facilitate input into the model. Each word is converted into a vector representation through an embedding layer. The word vector is processed using a Transformer architecture to capture the contextual information of the text. The feature vector generated by the text encoder is a semantic representation of the prompt text, which can effectively reflect the intention conveyed by the text. Selection of image encoder The CLIP image encoder is used in this method, which is also a model based on contrastive learning and can effectively extract image features. The image to be recognized (e.g., the initial image or other related images) is input into the image encoder. The image encoder converts the image into a feature vector, and specifically performs scaling, cropping, normalization, and other processing on the image to meet the model input requirements. A convolutional neural network (CNN) is used to extract local and global features of the image. The extracted features are fused through a fully connected layer or other mechanism to form a final image feature vector. The feature vector generated by the image encoder represents the visual information of the image and can effectively capture the important features in the image. The extracted prompt text feature vector and the feature vector of the image to be recognized will serve as the basis for subsequent processing. They will be used to construct the hand feature vector to ensure that the generated hand image is consistent with the prompt text description. By mapping text features and image features to the same feature space, the CLIP model can achieve alignment between text and image. This alignment helps the subsequent cross-attention processing, so that the model can effectively combine text information and image information to generate a target image that meets the requirements. Before constructing the hand feature vector, the prompt text data and the image to be recognized are encoded by the CLIP text encoder and the CLIP image encoder to generate corresponding feature vectors respectively. This process not only extracts basic information from text and images, but also lays the foundation for subsequent feature fusion and image generation, ensuring that the generated hand images can accurately reflect the text description and visual features.

[0037] In some embodiments of the present disclosure, constructing a hand feature vector according to a feature vector corresponding to the prompt text data includes: extracting hand parameters from the feature vector corresponding to the prompt text data through a gesture diffusion model; performing vector representation processing on the hand parameters to obtain a hand feature vector. For example, the gesture diffusion model can be a deep learning model specifically used to process hand-related tasks. The model can extract hand parameters from the input feature vector to help construct a more accurate hand feature vector. For example, the gesture diffusion model can be a MANO model, which is a popular hand parameter model that can generate high-quality hand shapes and postures. The MANO model describes the geometric features of the hand through a set of parameters (such as hand shape, posture, joint angles, etc.). When extracting hand parameters, the gesture diffusion model first analyzes the feature vector corresponding to the prompt text data to identify feature information related to the hand. These feature information include hand posture, finger bending degree, relative position of the hand and the object, etc. Through a deep learning algorithm, the model maps these feature information to specific hand parameters, such as finger length, width, joint angles, etc. The extracted hand parameters need to be processed by vector representation to form the final hand feature vector. This process usually includes: Standardizing the hand parameters to eliminate the dimensional differences between different parameters and ensure that they are compared on the same scale. Combining different hand parameters into a unified vector representation. For example, parameters such as finger length, width, and joint angle can be combined into a multidimensional vector. If the hand parameter dimension is too high, dimensionality reduction techniques (such as PCA, t-SNE, etc.) can be used to reduce the dimension while still retaining important information. The training process of the gesture diffusion model is based on the hand feature information in the prompt text data. The specific steps include: Collecting a large amount of prompt text data containing hand feature information and using the corresponding hand images as training samples. Using the hand feature information in the prompt text data, the model continuously adjusts the parameters through the back propagation algorithm so that the generated hand parameters can better match the real data. After multiple iterations, the model gradually optimizes its parameters and improves the extraction accuracy of hand features. After each iteration, the model evaluates the difference between the generated hand parameters and the real data and adjusts the parameters accordingly. After multiple iterations of training, the gesture diffusion model is able to learn the complexity of hand features and generate hand feature vectors with higher accuracy. This process not only improves the accuracy of the model, but also enhances its adaptability to different hand postures and shapes.

[0038] In some embodiments of the present disclosure, determining the initial image according to the feature vector corresponding to the prompt text data and the feature vector corresponding to the image to be recognized includes: processing the feature vector corresponding to the prompt text data and the feature vector corresponding to the image to be recognized by a cross-attention mechanism, and inputting the processing results into the initial diffusion model; processing the processing results by the initial diffusion model based on the noise map to obtain the initial image. For example, the feature vector is a way to convert text and image information into numerical representation. The feature vector of the prompt text data contains information related to hand features, and the feature vector of the image to be recognized reflects the visual features of the image. The combination of these two feature vectors can provide important information for generating the initial image. The cross-attention mechanism is a deep learning technology for processing information from different sources (such as text and images). For example, through the attention mechanism, the model's attention to important features can be enhanced. In this embodiment, the implementation process of the cross-attention mechanism is as follows: the feature vector corresponding to the prompt text data and the feature vector corresponding to the image to be recognized are used as input. Attention weights are generated by calculating the similarity between the two sets of feature vectors. These weights represent the correlation between text features and image features. Using the calculated attention weights, the feature vectors of the image to be recognized are weighted summed to obtain an image feature representation that integrates text information. The result of the cross-attention mechanism is input into the initial diffusion model. This processing result can be regarded as an enhanced feature representation, which contains the association information between text and image, and provides richer context for the subsequent generation of the initial image. The initial diffusion model is a generative model, which is usually used to generate images by stepwise denoising. Specifically, the initial diffusion model starts with a random noise image. This noise image can be randomly generated or a transformation of some preliminary image. Through multiple iterations, the noise image is gradually denoised. In each iteration, the model adjusts the content of the image according to the input enhanced feature representation (output from the cross-attention mechanism) to gradually approach the target image. After multiple denoising processes, the initial diffusion model generates an initial image. This image not only reflects the features of the image to be recognized, but also integrates the hand feature information in the prompt text data to ensure that the generated image meets the user's needs. The generated initial image can be quality checked through a certain evaluation mechanism. If the generated image fails to achieve the expected effect, it can be generated again by adjusting the input feature vector or optimizing the model parameters. In this embodiment, the feature vector corresponding to the prompt text data and the feature vector corresponding to the image to be recognized are processed through a cross-attention mechanism, and the results are input into the initial diffusion model. The present invention can effectively generate an initial image, providing a basis for subsequent image correction and optimization.

[0039] In some embodiments of the present disclosure, detecting an initial image to determine whether the initial image contains a hand region includes: inputting the initial image into a hand detection model, and identifying whether the initial image contains a hand detection frame through the hand detection model; if the initial image contains a hand detection frame, identifying the hand region in the initial image through the hand detection model. For example, the initial image is an image generated by an initial diffusion model. Before subsequent processing, it is necessary to confirm whether there is a hand region in the image. The hand detection model is a deep learning model for identifying the hand region in an image. The model can usually be based on a convolutional neural network (CNN) and can efficiently detect and locate the hand. The training of the hand detection model usually uses a large amount of annotated hand image data, and by learning the features of the hand, the hand region can be accurately identified in a new image. The generated initial image is input into the hand detection model, and the hand detection model processes the input initial image to identify the hand region contained therein. The specific processing flow includes: firstly, feature extraction is performed on the image to identify image features related to the hand. Through the learned features, each region in the image is classified to determine whether it is a hand. If a hand is detected, a corresponding bounding box will be generated to identify the position of the hand. If the hand detection model recognizes the hand region in the initial image, the hand detection frame is output. These detection frames usually contain the following information: the coordinates of the upper left corner and the lower right corner of the detection frame, indicating the specific position of the hand in the image. The width and height of the detection frame indicate the size of the hand region. After the hand region in the initial image is identified by the hand detection model, the region can be further processed, for example, the initial image is corrected or enhanced according to the detected hand region. More detailed feature information is extracted from the identified hand region for more accurate image generation or analysis. If the hand detection model fails to identify any hand detection frame, corresponding measures may need to be taken, for example, regenerating the image back to the initial diffusion model, adjusting the input parameters or feature vectors, and regenerating the image. Data enhancement processing is performed on the initial image to improve the recognition ability of the hand detection model. In this way, by inputting the initial image into the hand detection model, identifying and marking the hand region, the hand information in the image can be effectively judged, providing support for subsequent processing and optimization.

[0040] In some embodiments of the present disclosure, the feature vector corresponding to the prompt text data, the hand feature vector, and the feature vector corresponding to the hand region are fused to obtain the hand fusion feature vector, including: fusing the feature vector corresponding to the prompt text data and the hand feature vector to obtain a first fusion vector, and processing the first fusion vector through a multi-layer perceptron to obtain a first enhanced fusion vector; processing the feature vector corresponding to the hand region through a multi-layer perceptron, and linearly fusing the processing result with the first enhanced fusion vector through hyperparameters to obtain a hand fusion feature vector. For example, the feature vector corresponding to the prompt text data represents feature information related to the text description, which is usually generated by a natural language processing model (such as BERT, GPT, etc.). The hand feature vector represents the shape, posture and other related features of the hand, and is generated by a gesture diffusion model. The feature vector corresponding to the hand region represents the specific features of the hand region extracted from the initial image, which is usually obtained by an image processing model such as a convolutional neural network (CNN). By fusing these feature vectors, a comprehensive hand fusion feature vector is generated, which can better reflect the relationship between the text description and the hand features, and provide richer information for subsequent image generation or analysis. Specifically, the feature vector corresponding to the prompt text data is fused with the hand feature vector, which can usually be processed by simple splicing, weighted summation or other fusion methods. The specific steps are as follows: Splice the two feature vectors in dimension to form a new vector. Assign weights to each feature vector, and generate the first fusion vector by weighted summation. The first fusion vector obtained contains a comprehensive representation of text information and hand features. Input the first fusion vector into a multilayer perceptron for further processing. The multilayer perceptron consists of multiple fully connected layers (denselayers), which can perform nonlinear transformations on input features. Use activation functions (such as ReLU, Sigmoid, etc.) in each layer to enhance the expression ability of the model. After being processed by the multilayer perceptron, the first enhanced fusion vector is obtained. This vector enhances the original features to a certain extent and emphasizes important features. Input the feature vector corresponding to the hand area into another multilayer perceptron or the above multilayer perceptron for processing to generate a new feature representation. The purpose of this step is to extract the deep feature information of the hand area, making it more suitable for subsequent fusion operations. Fuse the hand area feature vector with the first enhanced fusion vector, using the hyperparameter linear fusion method. Specifically, two hyperparameters (e.g., α and β) are set to control the weights of the two feature vectors during the fusion process. Usually, the values ​​of these hyperparameters are between 0 and 1, and α+β=1 is satisfied. The hand fusion feature vector is calculated by the following formula:

[0041] Hand fusion feature vector = α·first enhanced fusion vector + β·hand region feature vector

[0042] The obtained hand fusion feature vector combines text information, hand features and hand region features, and can more comprehensively describe the features of the hand. The generated hand fusion feature vector can be used for subsequent image generation, gesture recognition or other visual tasks, providing more accurate context information, thereby improving the effect of the generation model.

[0043] Based on the above embodiment, by fusing the feature vector corresponding to the prompt text data, the hand feature vector and the feature vector corresponding to the hand area, a hand fusion feature vector can be effectively generated to provide richer feature information for subsequent image generation and analysis.

[0044] In some embodiments of the present disclosure, generating a target image based on an initial image, a feature vector corresponding to a hand region, a hand fusion feature vector, and a feature vector corresponding to prompt text data includes: performing cross-attention processing on the feature vector corresponding to the prompt text data to obtain a cross-attention feature vector; performing addition processing on the feature vector corresponding to the hand region and the hand fusion feature vector, and processing the result of the addition processing by a cross-attention mechanism to obtain a hand enhancement fusion vector; generating a target image based on the initial image, the cross-attention feature vector, and the hand enhancement fusion vector by a target diffusion model. For example, the initial image is an image generated by the above-mentioned initial diffusion model, which contains hand region and background information. The goal is to generate a target image based on the image that is more consistent with the description of the prompt text. The prompt text data is descriptive information related to the target image, and the feature vector is generated by a natural language processing model to represent the semantic features of the text. Cross-attention is a mechanism that can associate and fuse features between different modalities (such as text and image). The specific steps are as follows: Input the feature vector corresponding to the prompt text data into the cross-attention module. In this module, the feature vector interacts with the features of the initial image, calculates the attention weight, and emphasizes the important features related to the prompt text. The obtained cross-attention feature vector contains the correlation between the text information and the initial image features, which can better reflect the generation requirements of the target image. The feature vector corresponding to the hand area is added to the hand fusion feature vector, so that the specific features of the hand can be combined with the comprehensive features to enhance the representation ability of the hand. Through simple vector addition, a new feature vector is formed, which contains the specific information of the hand area and the fused context information. The result of the addition process is input into the cross-attention mechanism for further processing to enhance the expression of the hand features. The specific steps are as follows: In the cross-attention module, the attention weight between the added feature vector and the cross-attention feature vector is calculated. Through this mechanism, the model can dynamically adjust the relationship between the hand features and the text features, emphasize the important features related to the generation of the target image, and the obtained hand enhancement fusion vector integrates the correlation between the hand area features and the prompt text features, which can more accurately describe the features of the hand in the target image. The initial image, the cross-attention feature vector and the hand enhancement fusion vector are used as inputs of the target diffusion model to generate the target image. The specific steps are as follows: The initial image provides background information. The cross-attention feature vector provides semantic information of the text description. The hand enhancement fusion vector provides the specific features of the hand. In the target diffusion model, this information is combined to gradually generate a target image that matches the prompt text description in a step-by-step manner. The model uses a diffusion process to gradually denoise, starting with random noise and gradually introducing feature information until the final target image is generated. The generated target image can be used in a variety of application scenarios, such as virtual reality, game development, gesture recognition, etc.Through the feedback mechanism, the generated images can be evaluated and optimized to further improve the performance and generation quality of the model.

[0045] Based on the above embodiment, by performing cross-attention processing and addition processing on the initial image, the feature vector corresponding to the hand area, the hand fusion feature vector and the feature vector corresponding to the prompt text data, the target image can be effectively generated to ensure that the generated result meets the requirements of the text description and hand features.

[0046] In some embodiments of the present disclosure, the above-mentioned initial diffusion model is a basic diffusion model, which is mainly used to generate an initial image. It is usually a model pre-trained on a large-scale dataset, which aims to capture the basic features and structure of the image. The initial diffusion model learns how to recover the image from noise by gradually adding noise to the training image. This process includes forward diffusion (gradually adding noise) and backward diffusion (gradually removing noise). In the generation stage, the model starts with random noise and gradually generates a clear image through multiple backward diffusion steps. Each step uses the learning ability of the model to convert noise into image features. The initial diffusion model can generate a variety of images and is usually used to generate initial images with diversity and randomness. For example, it can generate hand images of different postures, shapes and backgrounds to provide a basis for subsequent processing. The above-mentioned target diffusion model is a fine-tuned diffusion model specifically used to generate target images that meet specific conditions (such as text descriptions and hand features). The model is usually fine-tuned for specific tasks based on the initial diffusion model. On specific tasks, the initial diffusion model is fine-tuned using labeled datasets (such as gestures, postures, etc.) to enhance its generation ability in specific contexts. By designing a loss function suitable for the task (such as adversarial loss, content loss, etc.), the generation quality of the model is optimized so that it can better meet the input conditions. The target diffusion model receives the initial image, the cross-attention feature vector, and the hand-enhanced fusion vector as input. These inputs together determine the content and style of the generated image. In the generation phase, the model generates the target image based on the above inputs by step-by-step denoising. The model combines the input features in each step and dynamically adjusts the generation process to ensure that the final image meets the requirements of the prompt text and hand features. The target diffusion model is suitable for generating high-quality images with specific requirements, such as gesture recognition, interactive images in virtual reality, etc. Through fine-tuning, the model can generate images that are more accurate and meet user needs. The initial diffusion model and the target diffusion model play different roles in image generation. The initial diffusion model provides the basic generation capability and can generate a variety of initial images; while the target diffusion model generates high-quality target images for specific task requirements through fine-tuning. This division of labor makes the entire image generation process more flexible and efficient, and can better meet the needs of different application scenarios.

[0047] refer to Figure 3 , the prompt text data is a text description with hand feature information, and the image to be recognized is a reference image with a hand area. The specific processing flow is as follows:

[0048] 1. CLIP text encoder: Input the text description into the CLIP text encoder to obtain text features;

[0049] 2. CLIP image encoder: Input the reference image into the CLIP image encoder to obtain image features;

[0050] 3. Basic diffusion model: The noisy image is input into the basic diffusion model. The text features and image features are fused into each module of the diffusion model through the cross attention mechanism. After multiple iterations, the basic image is generated.

[0051] 4. Gesture diffusion model: Input text features into the gesture diffusion model. Use MANO (a hand parameter model) or other hand learning models as a basis to reconstruct hand parameters based on text features. After multiple iterations, output gesture parameters.

[0052] 5. Hand detection: The image generated by the basic diffusion model is input into the hand detection model to identify the hand position in the image. The hand detection box is output to identify the hand area.

[0053] 6. Hand feature fusion generation: Input text features, gesture parameters and the area of ​​the hand detection box into the hand feature fusion module. Combine text features with gesture parameters and then pass through MLP to enhance the semantic understanding of hand movements. Pass MLP to linearly fuse the regional features within the hand detection box with the above combined features through hyperparameters to improve the feature expression of the hand area.

[0054] 7. Fine-tune the diffusion model: Input the image generated by the basic diffusion model into the fine-tune diffusion model. Fuse the text features into each module of the fine-tune diffusion model through the cross-attention mechanism. Add the features generated by the hand detection box and the hand feature fusion module, and fuse them into the model through the cross-attention mechanism. After multiple iterations, generate the final fused image.

[0055] Figure 4 It is a block diagram of an image text processing device provided by an embodiment of the present disclosure.

[0056] like Figure 4 As shown, the image text processing device 400 includes an acquisition module 410, a hand feature construction module 420, a first determination module 430, a second determination module 440, a third determination module 450, a fusion module 460 and a generation module 470.

[0057] Specifically, the acquisition module 410 is used to acquire prompt text data and an image to be recognized.

[0058] The hand feature construction module 420 is used to construct a hand feature vector according to a feature vector corresponding to the prompt text data when the prompt text data contains hand feature information.

[0059] The first determination module 430 is used to determine the initial image according to the feature vector corresponding to the prompt text data and the feature vector corresponding to the image to be recognized.

[0060] The second determination module 440 is used to detect the initial image to determine whether the initial image contains a hand area.

[0061] The third determination module 450 is used to determine a feature vector corresponding to the hand region when the initial image includes the hand region.

[0062] The fusion module 460 is used to fuse the feature vector corresponding to the prompt text data, the hand feature vector, and the feature vector corresponding to the hand area to obtain a hand fusion feature vector.

[0063] The generating module 470 is used to generate a target image based on the initial image, the feature vector corresponding to the hand area, the hand fusion feature vector, and the feature vector corresponding to the prompt text data.

[0064] The image text processing device 400 can effectively identify and extract hand features by acquiring prompt text data and the image to be identified and analyzing the hand feature information therein, thereby ensuring that the generated image is more in line with the user's expectations. By constructing a hand feature vector and fusing it with the prompt text data and the feature vector of the image to be identified, it is possible to integrate a variety of information and improve the accuracy and naturalness of hand image generation. An initial image detection step is introduced during the generation process to confirm the existence of the hand area. This process can ensure the rationality of the hand area in the generated target image and avoid generating unnatural or unrealistic hand postures. By fusing the hand area feature vector with other related feature vectors, the generated target image will be more realistic in terms of hand shape, posture and its interaction with objects, thereby significantly improving the overall quality of the image.

[0065] In some embodiments of the present disclosure, the hand feature construction module 420 is configured to extract hand parameters from a feature vector corresponding to prompt text data through a gesture diffusion model; perform vector representation processing on the hand parameters to obtain a hand feature vector.

[0066] In some embodiments of the present disclosure, the first determination module 430 is configured to: process the feature vector corresponding to the prompt text data and the feature vector corresponding to the image to be identified through a cross-attention mechanism, and input the processing results into the initial diffusion model; process the processing results through the initial diffusion model based on the noise map to obtain an initial image.

[0067] In some embodiments of the present disclosure, the second determination module 440 is configured to: input the initial image into a hand detection model, and identify whether the initial image contains a hand detection frame through the hand detection model; if the initial image contains a hand detection frame, identify the hand area in the initial image through the hand detection model.

[0068] In some embodiments of the present disclosure, the fusion module 460 is configured to: fuse the feature vector corresponding to the prompt text data and the hand feature vector to obtain a first fusion vector, and process the first fusion vector through a multi-layer perceptron to obtain a first enhanced fusion vector; process the feature vector corresponding to the hand area through a multi-layer perceptron, and linearly fuse the processing result with the first enhanced fusion vector through hyperparameters to obtain a hand fusion feature vector.

[0069] In some embodiments of the present disclosure, the generation module 470 is configured to: perform cross-attention processing on the feature vector corresponding to the prompt text data to obtain a cross-attention feature vector; perform addition processing on the feature vector corresponding to the hand area and the hand fusion feature vector, and process the result of the addition processing through the cross-attention mechanism to obtain a hand enhancement fusion vector; generate a target image based on the initial image, the cross-attention feature vector and the hand enhancement fusion vector through a target diffusion model.

[0070] In some embodiments of the present disclosure, before constructing a hand feature vector based on the feature vector corresponding to the prompt text data, the image text processing device 400 is also used to: encode the prompt text data through a text encoder to obtain a feature vector corresponding to the prompt text data; encode the image to be identified through an image encoder to obtain a feature vector corresponding to the image to be identified.

[0071] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules in the above-mentioned device embodiments are implemented.

[0072] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.

[0073] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0074] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0075] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0076] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0077] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A method for processing image text, characterized in that: include: Obtain prompt text data and the image to be recognized; When the prompt text data contains hand feature information, constructing a hand feature vector according to the feature vector corresponding to the prompt text data; Determine an initial image according to a feature vector corresponding to the prompt text data and a feature vector corresponding to the image to be recognized; Detecting the initial image to determine whether the initial image contains a hand area; When the initial image includes the hand region, determining a feature vector corresponding to the hand region; fusing the feature vector corresponding to the prompt text data, the hand feature vector, and the feature vector corresponding to the hand region to obtain a hand fusion feature vector; A target image is generated based on the initial image, the feature vector corresponding to the hand area, the hand fusion feature vector, and the feature vector corresponding to the prompt text data.

2. The method according to claim 1, characterized in that Constructing a hand feature vector according to the feature vector corresponding to the prompt text data includes: Extracting hand parameters from the feature vector corresponding to the prompt text data through a gesture diffusion model; The hand parameters are processed into vector representation to obtain the hand feature vector.

3. The method according to claim 1, characterized in that Determining the initial image according to the feature vector corresponding to the prompt text data and the feature vector corresponding to the image to be recognized includes: Processing the feature vector corresponding to the prompt text data and the feature vector corresponding to the image to be recognized through a cross attention mechanism, and inputting the processing result into the initial diffusion model; The processing result is processed by the initial diffusion model based on the noise map to obtain the initial image.

4. The method according to claim 1, characterized in that: Detecting the initial image to determine whether the initial image contains a hand region includes: Inputting the initial image into a hand detection model, and identifying whether the initial image contains a hand detection frame through the hand detection model; If the initial image includes a hand detection frame, the hand region in the initial image is identified by using the hand detection model.

5. The method according to claim 1, characterized in that The feature vector corresponding to the prompt text data, the hand feature vector, and the feature vector corresponding to the hand area are fused to obtain a hand fusion feature vector, including: Fusing the feature vector corresponding to the prompt text data and the hand feature vector to obtain a first fused vector, and processing the first fused vector through a multi-layer perceptron to obtain a first enhanced fused vector; The feature vector corresponding to the hand area is processed by the multi-layer perceptron, and the processing result is linearly fused with the first enhanced fusion vector through hyperparameters to obtain the hand fusion feature vector.

6. The method according to claim 1, characterized in that Generating a target image based on the initial image, the feature vector corresponding to the hand area, the hand fusion feature vector, and the feature vector corresponding to the prompt text data includes: Performing cross-attention processing on the feature vector corresponding to the prompt text data to obtain a cross-attention feature vector; Adding the feature vector corresponding to the hand region and the hand fusion feature vector, and processing the result of the addition by a cross attention mechanism to obtain a hand enhancement fusion vector; The target image is generated based on the initial image, the cross-attention feature vector and the hand-enhanced fusion vector through a target diffusion model.

7. The method according to claim 1, characterized in that Before constructing the hand feature vector according to the feature vector corresponding to the prompt text data, the method further includes: Encoding the prompt text data by a text encoder to obtain a feature vector corresponding to the prompt text data; The image to be identified is encoded by an image encoder to obtain a feature vector corresponding to the image to be identified.

8. An image text processing device, characterized in that: include: An acquisition module, used to acquire prompt text data and an image to be recognized; A hand feature construction module, used for constructing a hand feature vector according to a feature vector corresponding to the prompt text data when the prompt text data contains hand feature information; A first determination module, used for determining an initial image according to a feature vector corresponding to the prompt text data and a feature vector corresponding to the image to be identified; A second determination module is used to detect the initial image to determine whether the initial image contains a hand area; a third determining module, configured to determine a feature vector corresponding to the hand region when the initial image includes the hand region; A fusion module, used for fusing the feature vector corresponding to the prompt text data, the hand feature vector, and the feature vector corresponding to the hand area to obtain a hand fusion feature vector; A generation module is used to generate a target image based on the initial image, the feature vector corresponding to the hand area, the hand fusion feature vector, and the feature vector corresponding to the prompt text data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images