Image processing method and apparatus, computer device, storage medium and product

Abstract

The present application provides an image processing method and apparatus, a computer device, a storage medium and a product. The method comprises: obtaining an input image containing a preset object, and obtaining text data; performing encoding processing on the text data to obtain text embedding features; performing image feature extraction processing on the input image according to a plurality of predefined data dimensions, so as to obtain identity embedding features of the preset object under each data dimension; and, by means of using an interlaced condition mechanism, performing fusion and recognition processing on the identity embedding features of the preset object under each data dimension and the text embedding features to generate an output image comprising the preset object and comprising the features described by the text data. The present application can extract multi-dimensional identity features and, according to the interlaced condition mechanism, fuse the extracted multi-dimensional identity features and text features to generate personalized images, thus improving the accuracy of the personalized images.

Description

Image processing method, device, computer equipment, storage medium and product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 30, 2023, with application number 2023107985588 and application name “Image processing method, device, computer equipment, storage medium and product”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of computer technology, and in particular to an image processing method, an image processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0003] With the widespread application of artificial intelligence technology, image processing technology is pervasive in all aspects of daily life. Practice has shown that in order to meet the personalized needs of users, the demand for generating personalized images based on text data is growing.

[0004] Currently, the method for generating personalized images is usually to extract text features from text data and directly generate personalized images based on the text features. The personalized images generated by this method are relatively simple and not accurate enough.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide an image processing method, apparatus, computer equipment, storage medium, and product that can improve the accuracy of personalized images.

[0007] On the one hand, an embodiment of the present application provides an image processing method, which includes: obtaining an input image containing a preset object and obtaining text data; encoding the text data to obtain text embedding features; performing image feature extraction on the input image according to a plurality of predefined data dimensions to obtain identity embedding features of the preset object in each data dimension; calling a text generation image model to perform fusion recognition processing on the text embedding features of the preset object and the identity embedding features of each data dimension to generate an output image, wherein the text generation image model includes a plurality of cross-attention layers, and the text embedding features and the identity embedding features of each data dimension are alternately input into the cross-attention layers; the output image includes image features related to the preset object in the input image described by the text data.

[0008] On the one hand, an embodiment of the present application provides an image processing device, which includes: an acquisition unit, which is used to acquire an input image containing a preset object and acquire text data; a processing unit, which is used to encode the text data to obtain text embedding features; the processing unit is also used to perform image feature extraction processing on the input image according to multiple predefined data dimensions to obtain identity embedding features of the preset object in each data dimension; the processing unit is also used to call a text generation image model to perform fusion recognition processing on the text embedding features of the preset object and the identity embedding features of each data dimension to generate an output image, wherein the text generation image model includes multiple cross-attention layers, and the text embedding features and the identity embedding features of each data dimension are alternately input into the cross-attention layers; the output image includes image features related to the preset object in the input image described by the text data.

[0009] On the one hand, an embodiment of the present application provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the above-mentioned image processing method.

[0010] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned image processing method.

[0011] In one aspect, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image processing method.

[0012] In an embodiment of the present application, first, an input image and text data containing a preset object can be obtained, and the text data is used to describe the personalized features of the preset object; then the text data is encoded to obtain text embedding features; and image feature extraction is performed on the preset object in the input image according to multiple data dimensions to obtain the identity embedding features of the preset object in each data dimension; finally, the identity embedding features and text embedding features of the preset object in each data dimension are fused and recognized to generate a personalized image that matches the preset object. It can be seen that, on the one hand, when extracting the image features of the preset object, the present application can perform multi-dimensional feature extraction, so that the identity embedding features of the preset object can be extracted more comprehensively and accurately; on the other hand, in the process of generating personalized images based on identity embedding features and text features, the feature data is processed by designing multiple cross-attention layers in the text generation image model, which can be used to balance the conflict between identity embedding features and text embedding features, so that the contributions of different features are reasonably balanced during the image generation process, so that the generated output image can better meet the personalized needs described by the text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG1 is a schematic diagram showing the principle of an image processing solution provided in an embodiment of the present application;

[0014] FIG2 is a schematic diagram of the structure of an image processing system provided in an embodiment of the present application;

[0015] FIG3 is a flow chart of an image processing method provided in an embodiment of the present application;

[0016] FIG4 is a schematic diagram of an interface for obtaining text data provided in an embodiment of the present application;

[0017] FIG5 is a schematic diagram of an interface for extracting local identity embedding features provided by an embodiment of the present application;

[0018] FIG6 is a schematic diagram of a cross-adjustment process according to an embodiment of the present application;

[0019] FIG7 is a flow chart of another image processing method provided in an embodiment of the present application;

[0020] FIG8 is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0021] FIG9 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0022] FIG10 is a schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0024] The present application provides an image processing solution that can extract multi-dimensional identity features, and generate personalized images according to the fusion of multi-dimensional identity features and text features, which can improve the accuracy and efficiency of image processing. On the one hand, the present application can adopt an interlaced adjustment mechanism to perform interlaced conditional processing on global identity embedding features and text embedding features, which can avoid the problem of unbalanced feature contribution caused by the dominant factor of identity features; on the other hand, the present application can also adopt a local enhancement mechanism to enhance the local identity embedding features, so as to retain more local texture information of the user, thereby improving the accuracy of generating personalized images. Please refer to Figure 1, which is a schematic diagram of the principle of an image processing solution provided by an embodiment of the present application. The framework shown in Figure 1 mainly includes three encoders and a text-to-image synthesis network (hereinafter referred to as a text generation image model). The first encoder is a text encoder, which is responsible for converting the input text data into text embedding features (text embedding) p The second encoder is the global identity encoder, which is responsible for abstracting the specific identity information of the preset object in the input image (the preset object can refer to a pre-set object that needs to be personalized, such as some people, animals, etc.) into a global identity embedding feature (global identity embedding) globalThe third is a local texture encoder, which extracts hierarchical spatial embeddings (hereinafter referred to as local identity embeddings) from input images containing preset objects to preserve more texture details. After obtaining these embedding features (text embedding features, global identity embedding features, and local identity embedding features), the synthesis network is responsible for effectively fusing them together to generate personalized images that are both consistent with identity information and conform to the text description. During the generation of personalized images, on the one hand, the text embedding features and the global identity embedding features can be placed in the cross attention layer through an interlaced conditioning mechanism to avoid conflicts between text and identity control; on the other hand, the local identity embedding features can be fed into the local identity enhancement branch in the modified UNet decoder. This branch uses parallel mutual attention layers to adaptively fuse multiple levels of spatial embeddings. Based on the interlaced conditioning mechanism and the local enhancement mechanism, the personalized features described by the text data can be more accurately represented in the output image, improving the accuracy and generation efficiency of personalized images.

[0025] The principle of the image processing solution proposed in this application is briefly described below with reference to FIG1 :

[0026] ① To generate a personalized image, we first obtain an input image containing a user (pre-defined object). For example, the input image may include the user's face, head, limbs, and other parts. We also obtain text data describing the user's personalized characteristics, such as the text data (prompt) that reads: "A person wearing a red T-shirt." The input image and text data are then used to generate the personalized image.

[0027] ② The framework for generating personalized images shown in Figure 1 includes: three encoders (text encoder, global identity encoder, local texture encoder) and a text generation image model. Among them, the global identity encoder and the local texture encoder are used to extract image features of preset objects in the input image at different data dimensions, thereby obtaining identity embedding features of the preset objects in each data dimension. For example, the data dimensions may include: global dimensions and local dimensions. Specifically, the text encoder can be used to encode the text data to obtain text embedding features; the global identity encoder can be used to extract image features of the input image to obtain global identity embedding features; the local texture encoder can be used to extract image features of the input image to obtain local identity embedding features.

[0028] ③ Input the text embedding features, global identity embedding features, and local identity embedding features into the text-generated image model to generate a personalized image that matches the preset object. Specifically, an interleaved adjustment mechanism is used to interleave the text embedding features and the global identity embedding features, that is, the text embedding features and the global identity embedding features are interleaved and input into the cross-attention layer. For example, the text features are input into the first cross-attention layer, the global identity embedding features are input into the second cross-attention layer, the text features are input into the third cross-attention layer, the global identity embedding features are input into the fourth cross-attention layer, and so on. In addition, a local enhancement mechanism can also be used to enhance the local identity embedding features, that is, the local identity embedding features are input into the mutual attention layer for local feature enhancement. Finally, after recognition processing based on the text-generated image model, a personalized image that matches the preset object can be generated.

[0029] Next, we will introduce in detail the key technical terms involved in the image processing solution:

[0030] 1. Text data and personalized features:

[0031] Text data refers to data that uses text to describe the personalized features of a preset object in the personalized image to be generated. In other words, text data is usually a text description. The data format of the text data can be Chinese, English, a string, a code, etc. This application does not make specific restrictions on this. For example, text data can be expressed as: A person wearing a red T-shirt; for example, text data can also be expressed as: A person wearing a red T-shirt.

[0032] A personalized feature is a feature used to describe at least one of the following: clothing, makeup, behavior, or the environment in which the subject resides. For example, a personalized feature can be used to describe clothing features such as clothing color, style, and matching. Another example is a personalized feature that can be used to describe hair features such as makeup, hair color, and length. Another example is a personalized feature that can be used to describe the subject's physical characteristics, such as playing ball, exercising, or running.

[0033] 2. Identity Embedding Features

[0034] The identity embedding feature is a type of object image feature used to describe a preset object in different data dimensions. For example, if the data dimension is a global dimension, the identity embedding feature of the preset object in the global dimension can be expressed as a global identity embedding feature, which can be used to reflect the global identity information of the preset object, such as the position in the image, gender (male or female), age (child, teenager, elderly or adult), etc.; for another example, if the data dimension is a local dimension, the identity embedding feature of the preset object in the local dimension can be expressed as a local identity embedding feature, which can be used to reflect the local texture information of the preset object. For example, if the object to be processed includes the face of the preset object, the local identity embedding feature may include: eye features (single eyelids, double eyelids), mouth features (thick lips, cherry lips), skin type (dry skin, oily skin, combination dry skin, combination oily skin), etc.; for another example, if the object to be processed includes the head of the preset object, the local identity embedding feature may include: high skull, hair color (black, brown, red), hairstyle (curly hair or straight hair), long hair or short hair, etc.

[0035] 3. Staggered Adjustment Mechanism:

[0036] The interleaved adjustment mechanism is a mechanism for interleaved adjustment processing of two or more features. In the present application, the interleaved adjustment mechanism is used to interleave adjustment processing of the global identity embedding features and text embedding features of the preset object in the global dimension. The so-called interleaved adjustment processing refers to alternately inputting the global identity embedding features and the text embedding features into the cross-attention layer of the text generation image model, and interleaved adjustment processing of the global identity embedding features and the text embedding features based on each cross-attention layer; in other words, the interleaved adjustment mechanism can balance the difference between the global identity embedding features and the text embedding features in the process of generating personalized images. Specifically, this mechanism is applied to the cross-attention layer of the text generation image model, mainly to solve the problem that the global identity embedding features of the preset object are the dominant factor and the text embedding features lose control of the personalized image. This mechanism allows different conditions (i.e., text data) to be added independently without conflict.

[0037] 4. Personalized images:

[0038] A personalized image is an image generated based on text data that matches a preset object. Matching means being consistent with the preset object's identity characteristics, which can include any one or more of facial features, fingerprint features, hand features, and pupil features. In other words, a personalized image is generated by matching the preset object's personalized characteristics described in the text data. For example, if the input image contains person A and the text data represents "A person wearing a red T-shirt," the generated personalized image is an image of person A wearing a red T-shirt. Another example is if the input image contains person B and the text data represents "A man wearing a hat," the generated personalized image is an image of person B wearing a hat.

[0039] 5. Artificial Intelligence

[0040] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, large-scale image processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0041] Computer vision (CV) is the science of making machines "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, detecting, and measuring objects, and further processing them to produce images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has revolutionized the development of computer vision. Pre-trained models in the field of vision, such as the swin-transformer (a transformer), ViT (visual-related transformer), V-MOE (a visual architecture), and MAE (masked autoencoder), can be quickly and widely applied to specific downstream tasks through fine-tuning. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0042] The image processing scheme provided in this application mainly relates to computer vision technology in the field of artificial intelligence. Specifically, a pre-training model can be trained using computer vision technology, and the pre-training model can be used to generate an image model for text (such as the text-image synthesis network shown in Figure 1). Subsequently, the trained text-generated image model can be called to fuse and identify the identity embedding features and text embedding features of the preset object in the input image in each data dimension to generate a personalized image (i.e., output image) that matches the preset object. Among them, the so-called pre-training model (Pre-training model), also known as the cornerstone model or large model, refers to a deep neural network (DNN) with a large number of parameters, which is trained on a large amount of unlabeled data, and the function approximation ability of the large-parameter DNN is used to enable PTM to extract common features from the data. After fine-tuning, parameter efficient fine-tuning (PEFT), prompt-tuning and other technologies, it is suitable for downstream tasks (i.e., the pre-training model can be called to fuse and identify the identity embedding features and text embedding features of the preset object in each data dimension to generate a personalized image that matches the preset object). Therefore, pre-trained models can achieve ideal results in few-shot or zero-shot scenarios. PTMs can be divided into language models (ELMO, BERT, GPT), vision models (swin-transformer, ViT, V-MOE), speech models (VALL-E), and multimodal models (ViBERT, CLIP, Flamingo, Gato), depending on the data modality they process. Multimodal models refer to models that establish feature representations of two or more data modalities. Pre-trained models are important tools for outputting artificial intelligence generated content (AIGC) and can also serve as a universal interface for connecting multiple task-specific models.

[0043] 6. Cloud Technology

[0044] Cloud technology is a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Different levels of data will be processed separately, and data from various industries will require a strong system backend, which can only be achieved through cloud computing.

[0045] In the embodiments of the present application, text data is encoded to obtain text embedding features; image features of a preset object in an input image are extracted according to multiple data dimensions to obtain identity embedding features of the preset object in each data dimension; and an interleaved adjustment mechanism is adopted to fuse and identify the identity embedding features of the preset object in each data dimension and the text embedding features to generate a personalized image that matches the preset object. These processes all involve a large amount of data calculation and data storage services. The above processes require a large amount of computer operating costs. Therefore, the present application can implement related operation processes such as image processing and data screening based on cloud computing technology. Among them, cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" appear to users to be infinitely scalable and can be obtained at any time, used on demand, expanded at any time, and paid for on a per-use basis.

[0046] 7. Blockchain Technology

[0047] Blockchain is a new application model for computer technologies such as distributed data storage, peer-to-peer (P2P) transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database consisting of a series of data blocks (also called blocks) linked together using cryptographic methods. Each block contains information about a batch of online transactions, which is used to verify the validity of that information (to prevent counterfeiting) and generate the next block. Blockchain uses cryptography to ensure that data cannot be tampered with or forged.

[0048] In this application, the image processing process specifically involves: input image, text data, text embedding features, identity embedding features in various data dimensions, and personalized images, among other data. Optionally, this application can send this data to a blockchain for storage. The blockchain's immutability and traceability can prevent business data from being tampered with or leaked, thereby improving the data security and reliability of the image processing process.

[0049] It should be noted that in this application, the relevant data involved in the image processing process, such as: input image, text data, text embedding features, identity embedding features under various data dimensions, and personalized images, etc. When the above embodiments of this application are applied to specific products or technologies, user permission or consent must be obtained, and the relevant data collection, use and processing processes must comply with relevant national and regional laws, regulations and standards, and comply with the principles of legality, legitimacy and necessity, and do not involve obtaining data types prohibited or restricted by laws and regulations. In some optional embodiments, the relevant data involved in the embodiments of this application are obtained after the object has been separately authorized. In addition, when obtaining the object's separate authorization, the purpose of the relevant data involved is indicated to the object.

[0050] The image processing system proposed in the embodiments of the present application is introduced in detail below.

[0051] Please refer to Figure 2, which is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application. The architecture of the image processing system includes a server 204 and a terminal device cluster. The terminal device cluster includes multiple terminal devices, such as terminal device 201, terminal device 202, and terminal device 203. The number of terminal devices in the terminal device cluster is for illustrative purposes only and is not limited in the embodiments of the present application. Any terminal device in the terminal device cluster can be directly or indirectly connected to server 204 via wired or wireless communication.

[0052] Each terminal device in the terminal device cluster can be a mobile phone, a tablet computer, a laptop computer, a PDA, a mobile internet device (MID), a vehicle-mounted device, an aircraft, a wearable device (smart devices such as smart watches, smart bracelets, pedometers, etc.), a virtual reality device (such as VR (Virtual Reality) device, AR (Augmented Reality) device), etc. It can be understood that the type of each terminal device in the terminal device cluster can be the same or different. For example, terminal device 201 can be a mobile phone, and terminal device 202 can also be a mobile phone. For another example, terminal device 201 can be a tablet computer, and terminal device 203 can be a vehicle-mounted device. This application does not limit the number and type of terminal devices in the terminal device cluster.

[0053] Server 204 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0054] Next, taking any terminal device (e.g., terminal device 201) in the image processing system as an example, the interaction process between the terminal device 201 and the server 204 is described accordingly:

[0055] ① The terminal device 201 can obtain an input image containing a preset object and obtain text data. The text data is used to describe the personalized characteristics of the preset object.

[0056] ② Terminal device 201 can send input image and text data to server 204. Server 204 is equipped with a text encoder, a global identity encoder, a local texture encoder, and a text-to-image model. The text encoder encodes the text data; the global identity encoder and the local texture encoder extract identity embedding features of pre-set objects in the input image at different data dimensions.

[0057] ③ The server 204 may call a text encoder to perform encoder processing on the text data to obtain text features. Furthermore, the server 204 may perform image feature extraction processing on the preset object in the input image according to multiple data dimensions to obtain identity embedding features of the preset object in each data dimension; specifically, the server 204 may call a global identity encoder to perform image feature extraction processing on the preset object in the input image according to the global dimension to obtain the global identity embedding features (Global Identity Embedding) of the preset object; and, the server 204 may call a local texture encoder to perform image feature extraction processing on the preset object in the input image according to the local dimension to obtain the local identity embedding features of the preset object to achieve local information enhancement (Local Identity Enhancement).

[0058] ④ The server 204 performs a fusion recognition process on the text embedding features of the preset object and the identity embedding features under each data dimension to generate an output image, wherein the output image includes the image features related to the preset object in the input image described by the text data. The server 204 can adopt an interlaced adjustment mechanism to perform a fusion recognition process on the identity embedding features (global identity embedding features and local identity embedding features) and text embedding features of the preset object under each data dimension to generate a personalized image that matches the preset object. The interlaced adjustment mechanism is used to balance the difference between the identity embedding features and the text embedding features under any data dimension during the personalized image generation process. In one embodiment, the server 204 can call a text generation image model to perform a fusion recognition process on the text embedding features of the preset object and the identity embedding features of each data dimension to generate an output image, wherein the text generation image model includes multiple cross-attention layers, and the text embedding features and the identity embedding features of each data dimension are alternately input into the cross-attention layer.

[0059] ⑤ The server 204 sends the generated output image to the terminal device 201. The output image is a personalized image of the preset object in the input image.

[0060] It should be noted that the above-mentioned interactive process of image processing is only for example and does not limit the specific execution process of the terminal device and the server. Optionally, the text data is encoded to obtain text embedding features; and the image feature extraction process is performed on the preset object in the input image according to multiple data dimensions to obtain the identity embedding features of the preset object in each data dimension. The above process can also be performed by the terminal device; or, the text data is encoded to obtain text embedding features; and the image feature extraction process is performed on the preset object in the input image according to multiple data dimensions to obtain the identity embedding features of the preset object in each data dimension; the multiple cross-attention layers included in the text generation image model are used, and an interlaced adjustment mechanism is adopted to perform fusion recognition processing on the identity embedding features and text embedding features of the preset object in each data dimension to generate an output image that matches the preset object (i.e., a personalized image). The above process can also be performed separately by any terminal device or server in the image processing system.

[0061] In one possible implementation, the image processing system provided in the embodiment of the present application can be deployed on a node of a blockchain. For example, the server 204 and each terminal device included in the terminal device cluster (e.g., terminal device 201, terminal device 202, and terminal device 203, etc.) can be treated as node devices of the blockchain, together forming a blockchain network. Therefore, the image processing process of the first time-limited recognition model or the second time-limited recognition model in the present application can be executed on the blockchain, which can ensure the fairness and impartiality of the image processing process, while making the image processing process traceable and ensuring data security during the image processing process, thereby improving the security and reliability of the entire image processing process.

[0062] In an embodiment of the present application, the identity embedding features of the preset object can be extracted more comprehensively and accurately. In the process of generating personalized images based on the identity embedding features and text features, the text generation image model including multiple cross-attention layers is adopted. By utilizing the interleaved adjustment mechanism to balance the conflict between the identity embedding features and the text embedding features, the contribution of different features can be reasonably balanced in the image generation process, thereby making the generated personalized image more accurate.

[0063] It can be understood that the system architecture diagram described in the embodiment of the present application is intended to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0064] Next, specific embodiments of the image processing solution are described in detail with reference to the accompanying drawings.

[0065] Please refer to Figure 3, which is a flowchart of an image processing method provided by an embodiment of the present application. The image processing method can be executed by a computer device, which can be a terminal device or a server in the image processing system shown in Figure 2. The image processing method mainly includes but is not limited to the following steps S301 to S304:

[0066] S301: Obtain an input image containing a preset object and obtain text data. The text data is used to describe the personalized features of the preset object. The personalized features of the preset object described by the text data refer to: the text data is used to describe the features related to the preset object that are allowed to be included in the output image corresponding to the input image. That is to say, the features described by the text data can be displayed in the output image finally generated by S301 to S304. For example, if the text data is a person "wearing red clothes", and the personalized feature of "wearing red clothes" related to the person object (preset object) is described, then the person object wearing red clothes will be displayed in the personalized image after a series of processing from S302 to S304, and the feature of red clothes described in the text data will be included in the processed output image.

[0067] It should be noted that the preset objects mentioned in this application refer to objects such as people and animals that need to be personalized according to the characteristics described by the text data. The input image is the image to be processed, which needs to be processed to ultimately obtain an output image, which is a personalized image of the preset object.

[0068] In one possible implementation, a computer device obtains an input image containing a preset object, which may include any of the following: calling a shooting device to capture an image of the preset object to obtain an input image, that is, the input image is an image captured in real time; or, calling a shooting device to capture a video of the preset object to obtain a video, and selecting an input image from multiple images included in the video (for example, randomly selecting, or selecting an image with better image quality as the input image); or, obtaining an input image containing the preset object from an image database, that is, the input image is an image captured historically, etc. The embodiment of the present application does not specifically limit the method of obtaining the input image.

[0069] In one possible implementation, the text data may be acquired in real time or from a text database, which includes a plurality of historical text data that have been used to generate personalized images. The following describes the process of acquiring text data by taking the acquisition of text data in real time as an example. Please refer to FIG4 , which is a schematic diagram of an interface for acquiring text data provided by an embodiment of the present application. As shown in FIG4 , the text display interface S401 is provided with a data entry 4011. In response to the data entry being triggered (e.g., a double-click, long press, or other triggering operation), a text input panel S402 may be displayed. Preset objects may enter text data in the text input panel S402 according to business requirements. The text input panel S402 supports the entry of text data in multiple languages ​​and formats. For example, the entered text data may be: A person wearing a red T-shirt. Furthermore, after obtaining the text data, the computer device can also pre-process the text data. The pre-processing here may include at least one processing method such as data cleaning, normalization, format conversion (for example, English to Chinese conversion), etc. By adopting this method, by pre-processing the text data, it is beneficial to the subsequent encoding processing of the text data and improve the efficiency of image processing.

[0070] S302: Encode the text data to obtain text embedding features.

[0071] Specifically, the computer device may call a text encoder to encode the text data to obtain text embedding features. The text encoder may be a text processing model, which may be a natural language processing (NLP) model. The natural language processing model may include, but is not limited to, a Word2Vec (word embedding) model, a Transformer model, a Word2Vec (word embedding) model, a BERT (Bidirectional Encoder Representations from Transformers) model, and a GloVe (Global Vectors for Word Representation) model.

[0072] The following is a detailed description of the text data encoding process.

[0073] In one possible implementation, a computer device encodes text data to obtain text embedding features, specifically including the following steps: ① Segmenting the text data to obtain multiple text words, and extracting multiple keywords from the multiple text words in the text data. The purpose of segmenting the text data is to divide the text data into text words that are easier for the user to understand. For example, if the text data is "a man wearing a white T-shirt and black hair," the multiple text words obtained after segmenting the text data may include: a, wearing, white T-shirt, black hair, of, and man. Additionally, a keyword algorithm may be used to extract keywords from the multiple text words. For example, the extracted keywords may include: wearing, white T-shirt, black hair, and man.

[0074] ② Call the text encoder to perform feature extraction on each extracted keyword to obtain a word vector for each keyword. Specifically, a text processing model can be used to perform word embedding processing on each keyword. Examples of text processing models include, but are not limited to, the Word2Vec (word embedding) model, the BERT (Bidirectional Encoder Representations from Transformers) model, and the GloVe (Global Vectors for Word Representation) model. These models can be used to extract word vectors for each keyword.

[0075] ③ Generate text embedding features based on the extracted word vectors. Specifically, the word vectors of the extracted keywords can be combined into text embedding features. For example, the feature dimension of a word vector is 1*256. If the text data includes 256 keywords, the feature dimension of the text embedding feature is 256*256.

[0076] In this implementation, a text encoder (text processing model) can be called to perform accurate and efficient feature extraction on text data, thereby making the extracted text embedding features more accurate.

[0077] S303: performing image feature extraction processing on the input image according to a plurality of predefined data dimensions to obtain identity embedding features of the preset object in each data dimension.

[0078] Specifically, different data dimensions are used to indicate the extraction and processing of the image features of the preset object at different levels, that is, the identity embedding features under different data dimensions are used to reflect the image features of the preset object at different levels in the input image. In an embodiment of the present application, the data dimensions include global dimensions and local dimensions. The so-called global dimension is used to indicate that the identity features of the preset object are extracted from the global and overall level. For example, the user's gender, age and other features can be expressed as identity embedding features under the global dimension; the so-called local dimension is used to indicate that the identity features of the preset object are extracted from the local and detailed level. For example, the user's face, eyes and other features can be expressed as identity embedding features under the local dimension. The extraction process of the identity embedding features of the preset object in the global dimension (also called global identity embedding features) and the identity embedding features of the preset object in the local dimension (also called local identity embedding features) are described in detail below.

[0079] ① Extract the global identity embedding features of the preset object according to the global dimension.

[0080] In one possible implementation, a computer device calls a global identity encoder to perform identity recognition processing on a preset object in an input image according to a global dimension, and obtains a global identity embedding feature of the preset object. The global identity embedding feature is used to reflect the global identity information of the preset object, such as its position in the image, gender (male or female), age (child, teenager, elderly or adult), and other features. The global identity encoder is an image processing model, which may include, but is not limited to, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, and a feedforward neural network (FNN) model.

[0081] In a specific implementation, the global identity encoder includes: a first image sub-encoder and a second image sub-encoder. The first image sub-encoder is an encoder with non-adjustable parameters, such as a frozen CLIP image encoder (Contrastive Language-Image Pre-Training, a visual model with strong transferability obtained by training based on text as a supervisory signal), and the second image sub-encoder is an encoder with adjustable parameters, such as an image classification model (CNN model). The computer device calls the global identity encoder to perform identity recognition processing on a preset object in the input image according to a global dimension to obtain a global identity embedding feature of the preset object. The steps may include: ① Calling the first image sub-encoder to perform image encoding processing on the preset object contained in the region-adjusted image to obtain a first image feature of the preset object. The first image feature is used to reflect the attribute characteristics of the preset object. For example, the attribute characteristics may include but are not limited to: location attributes (horizontal and vertical coordinates, longitude and latitude, etc.), gender attributes (male or female), and age attributes (children, teenagers, elderly people). ② Calling the second image sub-encoder to perform image encoding processing on the preset object contained in the region-adjusted image to obtain a second image feature of the preset object. The second image feature is used to reflect the classification characteristics of the preset object. For example, the classification characteristics may include the object type of the preset object (human, cat, dog). ③ Based on the first image feature and the second image feature, a global identity embedding feature of the preset object is obtained. For example, the first image feature and the second image feature can be averaged and the averaged feature can be used as the global identity embedding feature; another example is that the first image feature and the second image feature can be weighted and the weighted feature can be used as the global identity embedding feature; another example is that the first image feature and the second image feature can be aligned and the aligned first image feature and the aligned second image feature can be concatenated to obtain the global identity embedding feature. The so-called feature alignment refers to unifying the first image feature and the second image feature into the same feature dimension.

[0082] Optionally, the computer device calls the global identity encoder and performs identity recognition processing on the preset object in the input image according to the global dimension. Before obtaining the global identity embedding feature of the preset object, the following steps are also included: first, the preset object in the input image is detected to obtain a detection result. For example, a detector can be used to detect the preset object in the input image, and the edge of the detected preset object can be extracted to obtain a detection result. The detection result includes a detection box (such as a rectangular box, an elliptical box, a polygonal box, etc.). Then, the input image is segmented according to the detection result to obtain a regional image containing the preset object; wherein, the regional image refers to the image within the detection box containing the preset object. Finally, the spatial resolution of the global identity encoder can be obtained, and the regional image can be adjusted according to the spatial resolution to obtain a regional adjusted image. wherein the image adjustment processing includes: adjusting the resolution or any one or more of image enhancement. For example, the image resolution of the regional image can be adjusted to the spatial resolution of the global identity encoder, so that the identity embedding features of the preset object in the input image can be easily extracted; for example, in the process of image adjustment processing of the regional image, it also includes any one or both of the clarity adjustment and image size adjustment of the regional image, and the regional image can be subjected to image enhancement processing (for example, improving the clarity of the regional image, image compression, adjusting the image size, etc.), so that the identity embedding features of the preset object can be extracted more accurately. In this way, the global identity encoder can be more focused on extracting the identity information of the preset object, so that the extracted identity embedding features can better represent the identity features of the preset object in the input image, thereby improving the accuracy of the global identity embedding features. In one possible embodiment, the spatial resolution can be set in the global identity encoder to adjust the image resolution according to the specified spatial resolution.

[0083] In the above implementation, a CLIP image encoder with fixed parameters and an image classification model with adjustable parameters are used together to extract global identity embedding features for a preset object. This approach has two advantages: first, the CLIP image encoder's powerful prior knowledge, obtained from a wide range of text-image pairs, can be leveraged; second, the first image features obtained from the CLIP image encoder can share the same semantic space as the CLIP text embedding features. To encode multiple layers of semantic information into the text embedding features, hierarchical semantic features (second image features) can be extracted from a learnable image classification model (e.g., a ConvNeXt model) with rich semantic prior knowledge. The extracted second image features are projected into a single vector through multiple fully connected layers (FCLs), and multiple first image features are concatenated, making the final extracted global identity embedding features more accurate and reliable.

[0084] ② Extract the local identity embedding features of the preset object according to the local dimension.

[0085] In one possible implementation, a computer device calls a local texture encoder to perform feature extraction processing on a preset object in an input image according to a local dimension to obtain a local identity embedding feature of the preset object. The local identity embedding feature is used to reflect the local texture information of the preset object. For example, if the object to be processed includes the face of the preset object, the local identity embedding feature may include facial features such as eye features (single eyelids, double eyelids), mouth features (thick lips, cherry lips), and skin type (dry skin, oily skin, mixed dry skin, mixed oily skin). For another example, if the object to be processed includes the head of the preset object, the local identity embedding feature may include features such as a high skull, hair color (black, brown, red), hairstyle (curly hair or straight hair), and long hair or short hair. The local texture encoder is also an image processing model, which may include, but is not limited to, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, and a feedforward neural network (FNN) model. It should be noted that the model structures of the global identity encoder and the local identity encoder may be the same or different, and the embodiments of the present application do not impose any specific restrictions on this. In addition, the feature dimensions of the global identity embedding feature and the feature dimensions of the local identity embedding feature may be the same or different, and the embodiments of the present application also do not impose any specific restrictions on this.

[0086] Please refer to Figure 5, which is a schematic diagram of an interface for extracting local identity embedding features provided by an embodiment of the present application. As shown in Figure 5, when performing local identity embedding feature extraction, an input image containing a preset object can be displayed in the image display interface S501, where the input image contains the head of the preset object, and a circle tool 5011 is provided in the image display interface S501. The circle tool is used to trigger the highlighting of the circled part of the preset object. For example, the user clicks the circle tool 5011 in the image display interface S501 to circle the nose of the preset object, and then the nose is enlarged and displayed in the image display interface S502 (as shown in 5021 in the interface S502). Then, as the nose is enlarged, more local features of the nose can be displayed in the image display interface S502, for example, the image display interface S502 displays: a wide nose, a bulbous nose, and large nose wings.

[0087] By adopting the above implementation method, in the process of extracting the global identity embedding features of the preset object, the key parts of the preset object can be highlighted, so as to accurately extract the features of the highlighted key parts to extract more detailed local identity embedding features, making the extracted local identity embedding features more accurate and rich.

[0088] S304: Calling a text-generated image model to perform fusion recognition processing on the text embedding features of the preset object and the identity embedding features of each data dimension, and generate an output image that includes the preset object and includes the features described by the text data, wherein the text-generated image model includes a plurality of cross-attention layers, and the text embedding features and the identity embedding features of each data dimension are alternately input into the cross-attention layers, and the output image includes image features related to the preset object in the input image described by the text data. In one embodiment, the text-generated image model including the cross-attention layer utilizes an interlaced adjustment mechanism to perform fusion recognition processing on the identity embedding features and text embedding features of the preset object in each data dimension, and finally generates an output image that includes the preset object and includes the features described by the text data, thereby realizing personalized image processing of the preset object.

[0089] In specific implementation, an interleaved adjustment mechanism is adopted, and the text-generated image model can be called to perform fusion recognition processing on the identity embedding features and text embedding features of the preset object in various data dimensions to generate a personalized image that matches the preset object. The interleaved adjustment mechanism is used to balance the difference between the identity embedding features and the text embedding features in any data dimension during the personalized image generation process. The difference here refers to the difference between the feature contributions made by the identity embedding features (such as the global identity embedding features) and the text embedding features in any data dimension to the text-generated image model during the personalized image generation process. In other words, the interleaved adjustment mechanism can reasonably balance the identity embedding features and the text embedding features to eliminate the problem that the global identity embedding features of the preset object dominate and the text embedding features lose control of the personalized image, avoid conflicts between features, and improve the accuracy of personalized images.

[0090] In one possible implementation, the identity embedding feature includes: a global identity embedding feature and a local identity embedding feature; the computer device adopts an interlaced adjustment mechanism to fuse and identify the identity embedding features of the preset object in each data dimension and the text embedding features to generate a personalized image that matches the preset object. Specifically, the following steps may be included: first, an interlaced adjustment mechanism is adopted to call the text generation image model to perform interlaced adjustment on the global identity embedding features and text embedding features of the preset object to obtain a first feature result; a local enhancement mechanism is adopted to call the text generation image model to perform feature enhancement on the local identity embedding features of the preset object to obtain a second feature result; next, the text generation image model is called to decode and identify the first feature result and the second feature result to generate a personalized image that matches the preset object. Specifically, the text generation image model in this application is an improved UNet Denoiser. As shown in Figure 1, the UNet Denoiser includes two parts, an encoder and a decoder. The encoder part is composed of multiple cross-attention layers, and the decoder part is composed of multiple cross-attention layers and multiple parallel mutual attention layers. It should be noted that the interleaved adjustment mechanism can be applied to the cross-attention layer in both the encoder and decoder parts; the local enhancement mechanism is applied to the mutual attention layer in the decoder part.

[0091] The specific implementation process of the staggered regulation mechanism and the specific implementation process of the local enhancement mechanism will be explained in detail below.

[0092] (1) Interlaced Condition Mechanism:

[0093] In one possible implementation, the text generation image model includes k cross attention layers, any cross attention layer is represented as the jth cross attention layer, k and j are both positive integers and j≤k. The computer device calls the text generation image model to perform cross adjustment processing on the global identity embedding feature and the text embedding feature of the preset object according to the cross adjustment mechanism to obtain a first feature result, which specifically includes the following steps: first, the global identity embedding feature is input into the jth cross attention layer, and the text embedding feature is input into the j+1th cross attention layer; until the k cross attention layers are all input with the global identity embedding feature or the text embedding feature, the global identity embedding feature and the text embedding feature are respectively cross adjusted based on the k cross attention layers to obtain the first feature result. Please refer to Figure 6, which is a flow chart of a cross adjustment process provided in an embodiment of the present application. As shown in Figure 6, first, the text embedding features can be input into the cross-attention layer 1, and the global identity embedding features can be input into the cross-attention layer 2; then the text embedding features can be input into the cross-attention layer 3, and the global identity embedding features can be input into the cross-attention layer 4, and so on. The text embedding features and the global identity embedding features can be alternately input into the k cross-attention layers. When each cross-attention layer is input with text embedding features or global identity embedding features, the text embedding features and the global identity embedding features can be interlaced and adjusted based on these k cross-attention layers.

[0094] In a specific implementation, any cross-attention layer with a global identity embedding feature as input is denoted as Si, and any cross-attention layer with a text embedding feature as input is denoted as Sj. The computer device performs interlaced adjustment processing on the global identity embedding feature and the text embedding feature based on k cross-attention layers to obtain a first feature result, which specifically includes the following steps: first, based on each cross-attention layer Si in the k cross-attention layers, attention extraction processing is performed on the global identity embedding feature to obtain the attention feature of the global identity embedding feature; then, based on each cross-attention layer Sj in the k cross-attention layers, attention extraction processing is performed on the text embedding feature to obtain the attention feature of the text embedding feature; finally, the attention feature of the global identity embedding feature and the attention feature of the text embedding feature are fused to obtain the first feature result. The first feature result refers to the result of attention extraction processing on the text embedding feature and the global identity embedding feature, and the first feature result is used to reflect the key information in the text data and the important identity feature information of the preset object.

[0095] Based on the staggered adjustment mechanism, the problem that user identity features dominate and text embedding features lose control of personalized images can be solved. This mechanism allows different conditions to be added independently without conflict, thereby balancing the differences between global identity embedding features and local identity embedding features.

[0096] (2) Local Identity Enhancement:

[0097] In one possible implementation, a text-to-image model includes a parallel mutual attention layer and a self-attention layer. The parallel mutual attention layer utilizes a parallel attention mechanism for feature processing, while the self-attention layer utilizes a self-attention mechanism for feature processing. A computer device invokes the text-to-image model to perform feature enhancement processing on the local identity embedding features of a preset object according to a local enhancement mechanism, thereby obtaining a second feature result. Specifically, the computer device includes the following steps: ① Obtaining a background embedding feature of an input image, which is obtained by performing image recognition on the background image excluding the preset object in the input image; specifically, invoking an image processing model to perform image recognition processing on the background image to obtain the background embedding feature. The image processing model may include, but is not limited to, a CNN model, an RNN model, or an FNN model. ② Invoking the self-attention layer to recognize the background embedding feature and obtain a self-attention recognition result; and ③ Invoking the mutual attention layer to recognize the local identity embedding features of the preset object and obtain a mutual attention recognition result. ③ In accordance with the local enhancement mechanism, performing feature enhancement processing on the self-attention recognition result and the mutual attention recognition result to obtain a second feature result. For example, the self-attention recognition result and the mutual attention recognition result can be added to obtain the final second feature result, which can be used to reflect the feature result after local information enhancement.

[0098] In a specific implementation, the local identity embedding feature includes multiple spatial embedding features; any two spatial embedding features are represented as: Yk and Yv. The computer device invokes a mutual attention layer to identify and process the local identity embedding features of a preset object, obtaining a mutual attention recognition result. The steps include: first, using the spatial embedding feature Yk as the key of the mutual attention layer, and using the spatial embedding feature Yv as the value of the mutual attention layer; then, obtaining the mutual attention weights of the mutual attention layer, and fusing the spatial embedding features included in the local identity embedding feature according to the mutual attention weights, to obtain a mutual attention result. The mutual attention layer follows the mutual attention mechanism, while the self-attention layer follows the self-attention mechanism. Both the mutual attention mechanism and the self-attention mechanism are types of attention mechanisms. The so-called attention mechanism refers to expressing each word in terms of other words in the sentence. In other words, the attention mechanism is a mechanism for capturing relationships between objects, and different words in the other words have different weights for expressing the word. The mutual attention mechanism is also called the multi-headed attention mechanism. The so-called "multi-headed" attention mechanism, i.e., multiple attention mechanisms, means that it can capture relationships at different levels of abstraction from different perspectives. In this application, if the local identity embedding feature includes n (n is a positive integer) spatial embedding features, for any spatial embedding feature, it can be expressed based on the other n-1 spatial embedding features. In this way, more accurate and rich texture information can be obtained from the extracted local identity embedding features.

[0099] For example, the two sets of spatial embedding features (multi-level spatial embedding) obtained by the local texture encoder can be expressed as follows: and And use this as the key (K) and value (V) to be put into the parallel mutual attention layer. The operation in the mutual attention layer will be expressed as the following formula (1) (superscripts are omitted for simplicity):

[0100] In the above formula (1), A learnable projection matrix for the text-to-image model, which includes the attention weights of each spatial embedding feature; Q L Refers to the query matrix in the mutual attention mechanism, that is, the important image features that need to be paid attention to in the mutual attention mechanism; K l refers to the key matrix in the mutual attention mechanism; V l Refers to the value matrix in the mutual attention mechanism; d refers to the above Query matrix, key matrix, value matrix (i.e. Q L , K l 、V l). This formula allows the identity features of local matches to be integrated into the text generation image model based on the attention weights, thereby achieving better identity preservation.

[0101] Next, the self-attention recognition result and the mutual attention recognition result are added together to obtain the second feature result. The calculation process is shown in the following formula (2): Output = λ × attention (Q l ,K l ,V l )+attention(Q ori ,K ori ,V ori ) (2)

[0102] In the above formula (2), attention(Q l ,K l , V l ) is the mutual attention recognition result, Q L , K l 、V l They refer to the query matrix, key matrix, and value matrix in the mutual attention mechanism respectively; attention(Q ori ,K ori ,V ori ) is the self-attention recognition result, Q ori , K ori 、V ori They refer to the query matrix, key matrix, and value matrix in the self-attention mechanism respectively; λ is a hyperparameter. Based on formula (2), the self-attention recognition result and the mutual attention recognition result can be added to obtain the second feature result.

[0103] Based on the local enhancement mechanism, during the image generation process, the present application obtains rich texture information from the reference spatial features. On the basis of the self-attention layer in the text-generated image model, a new parallel mutual attention layer can be introduced to embed local spatial information (i.e., local identity embedding features), thereby extracting more texture details in the local identity embedding features, providing a more accurate prerequisite for the personalized image generation process, thereby generating personalized images more accurately.

[0104] In an embodiment of the present application, multi-dimensional image feature extraction of a preset object can be performed, so that the identity embedding features of the preset object can be extracted more comprehensively and accurately; and, in the process of generating personalized images based on identity embedding features and text features, the adopted interleaving adjustment mechanism can be used to balance the conflict between the identity embedding features and the text embedding features, so that the contributions of different features are reasonably balanced during the image generation process, thereby making the generated personalized image more accurate.

[0105] Please refer to Figure 7, which is a flowchart of another image processing method provided by an embodiment of the present application. The image processing method can be executed by a computer device, which can be a terminal device or a server in the image processing system shown in Figure 2. The image processing method mainly includes but is not limited to the following steps S701 to S707:

[0106] S701: Acquire an input image containing a preset object and obtain text data. The text data is used to describe features related to the preset object that are allowed to be included in an output image corresponding to the input image. In other words, the text data is used to describe personalized features of the preset object so that the personalized features can be displayed in the final processed output image.

[0107] In one possible implementation, the input image can be an image captured in real time, an image obtained from a real-time video, or an image captured historically, which is not specifically limited in this application. The text data can be data entered in real time, or obtained from a text database that includes multiple historical text data that have been used to generate personalized images.

[0108] S702: Encode the text data to obtain text embedding features.

[0109] In one possible implementation, a computer device encodes text data to obtain text embedding features, specifically including the following steps: ① Performing word segmentation on the text data to obtain multiple text words, and extracting multiple keywords from the multiple text words in the text data. The word segmentation process is intended to divide the text data into text words that are easier for the user to understand. For example, if the text data is: "A man wearing a white T-shirt and black hair," the multiple text words obtained after word segmentation of the text data may include: "a man, wearing, white T-shirt, black hair, of, man." Alternatively, a keyword algorithm may be used to extract keywords from the multiple text words. For example, the extracted keywords may include: "wearing, white T-shirt, black hair, man." ② Calling a text encoder to perform feature extraction on each extracted keyword to obtain a word vector for each keyword. Specifically, a text processing model can be used to embed each keyword. Examples of such models include, but are not limited to, the Word2Vec (word embedding) model, the BERT (Bidirectional Encoder Representations from Transformers) model, and the GloVe (Global Vectors for Word Representation) model. Using these models, word vectors for each keyword can be extracted. (3) Based on the extracted word vectors, text embedding features are generated.

[0110] S703: Calling a global identity encoder to perform identity recognition processing on a preset object in the input image according to a global dimension to obtain a global identity embedding feature of the preset object.

[0111] In one possible implementation, a computer device invokes a global identity encoder to perform identity recognition processing on a preset object in an input image according to a global dimension to obtain a global identity embedding feature of the preset object. This may include the following steps: ① Invoking a first image sub-encoder to perform image encoding processing on the preset object contained in the region-adjusted image to obtain a first image feature of the preset object. The first image feature is used to reflect attribute characteristics of the preset object. For example, the attribute characteristics may include, but are not limited to, location attributes (horizontal and vertical coordinates, longitude and latitude, etc.), gender attributes (male or female), and age attributes (child, teenager, elderly). ② Invoking a second image sub-encoder to perform image encoding processing on the preset object contained in the region-adjusted image to obtain a second image feature of the preset object. The second image feature is used to reflect classification characteristics of the preset object. For example, the classification characteristics may include the object type of the preset object (human, cat, dog). ③ Based on the first image feature and the second image feature, a global identity embedding feature of the preset object is obtained. For example, feature fusion processing is performed on the first image feature and the second image feature to obtain a global identity embedding feature. The feature fusion processing may include at least any of the following: feature weighting processing, feature alignment processing, and feature operation processing.

[0112] S704: calling a local texture encoder to perform feature extraction processing on a preset object in the input image according to a local dimension to obtain a local identity embedding feature of the preset object.

[0113] In a specific implementation, the local texture encoder is also an image processing model, which may include, but is not limited to, a Convolutional Neural Network (CNN) model, a Recurrent Neural Network (RNN) model, and a Feedforward Neural Network (FNN) model. It should be noted that the model structures of the global identity encoder and the local identity encoder may be the same or different, and this embodiment of the present application does not specifically limit this. In addition, the feature dimensions of the global identity embedding feature and the local identity embedding feature may be the same or different, and this embodiment of the present application also does not specifically limit this.

[0114] It should be noted that the specific process of extracting the global identity embedding features and local identity embedding features of the preset object in the embodiment of this application can be referred to the relevant process in the embodiment of Figure 3, and the embodiment of this application will not be repeated here. In addition, there is no restriction on the order of extracting the text embedding features, global identity embedding features, and local identity embedding features. For example, the text embedding features can be extracted first, and then the global identity embedding features and local identity embedding features can be extracted at the same time; for example, the global identity embedding features can be extracted first, then the local identity embedding features can be extracted, and finally the text embedding features can be extracted, and so on.

[0115] S705: Calling the text-to-image model to perform interlaced adjustment processing on the global identity embedding feature and the text embedding feature of the preset object to obtain a first feature result.

[0116] In one possible implementation, the text-generated image model includes k cross-attention layers, any one of which is represented as the jth cross-attention layer, where k and j are both positive integers and j≤k. The computer device, in accordance with the interleaved adjustment mechanism, calls the text-generated image model to perform interleaved adjustment processing on the global identity embedding feature and the text embedding feature of a preset object to obtain a first feature result, specifically comprising the following steps: first, inputting the global identity embedding feature into the jth cross-attention layer, and inputting the text embedding feature into the j+1th cross-attention layer; until all k cross-attention layers are input with the global identity embedding feature or the text embedding feature, interleaved adjustment processing is performed on the global identity embedding feature and the text embedding feature based on the k cross-attention layers to obtain the first feature result.

[0117] S706: Calling the text-to-image model to perform feature enhancement processing on the local identity embedding feature of the preset object to obtain a second feature result.

[0118] In one possible implementation, a computer device calls a text-generated image model to perform feature enhancement processing on the local identity embedding features of a preset object according to a local enhancement mechanism to obtain a second feature result, specifically including the following steps: First, the background embedding features of the input image can be obtained, and the background embedding features are obtained after image recognition of the background image other than the preset object in the input image. Then, the self-attention layer is called to perform recognition processing on the background embedding features to obtain a self-attention recognition result; and the mutual attention layer is called to perform recognition processing on the local identity embedding features of the preset object to obtain a mutual attention recognition result. Finally, according to the local enhancement mechanism, feature enhancement processing is performed on the self-attention recognition result and the mutual attention recognition result to obtain a second feature result. Among them, the text-generated image model adopted in the embodiment of the present application may include, but is not limited to: a Stablediffusion model and an Imagen model. The present application does not specifically limit the model structure of the text-generated image model.

[0119] S707: Calling the text-to-image model to decode and recognize the first feature result and the second feature result to generate an output image, wherein the output image includes a preset object and features described by the text data. The text-to-image model includes multiple cross-attention layers, into which text embedding features and identity embedding features of various data dimensions are alternately input; the output image includes image features described by the text data and related to the preset object in the input image.

[0120] In specific implementation, the first feature result refers to the result after attention extraction processing of the text embedding feature and the global identity embedding feature. The first feature result is used to reflect the key information in the text data and the important identity feature information of the preset object; the second feature result is used to reflect the feature result after local information enhancement. Therefore, the first feature result and the second feature result are obtained after feature processing of the text embedding feature, the global identity embedding feature, and the local identity embedding feature using an interlaced adjustment mechanism and a local enhancement mechanism. Subsequently, after calling the decoder in the text generation image model to decode and identify the first feature result and the second feature result, a personalized image can be generated. The personalized image can not only maintain identity consistency with the preset object in the input image, but also have the personalized features described by the text data, thereby meeting the personalized needs of image generation.

[0121] Practice shows that the comparison results of the personalized image generation method provided by the embodiment of the present application and other traditional methods are shown in Table 1 below:

[0122] Table 1. Comparison results between this application and other traditional methods

[0123] As can be seen from the above table, the image generation method provided in the embodiment of the present application is superior to the traditional method in terms of both identity consistency (Identity Similarity) and text alignment (Text Alignment), which shows that the personalized image generation method provided by the present application has high accuracy; in addition, the method provided by the present application also greatly shortens the processing time, which shows that the personalized image generation method provided by the present application has high efficiency.

[0124] It should be understood that the above image processing process can be applied to various personalized image generation scenarios for business needs. The image processing scenarios provided in the embodiments of the present application are described below.

[0125] Please refer to Figure 8, which is a schematic diagram of an image processing scenario provided by an embodiment of the present application. As shown in Figure 8, the image processing scenario involves a terminal device and a server. The terminal device can be a device used by a preset object, and the server is configured with a text-to-image model for performing image processing on input images and text data uploaded by the terminal device. The specific process of generating personalized images is as follows: ① The user can upload an input image containing a preset object and text data (for example: A woman wearing a red t-shirt) through a terminal device, and then initiate an image processing request to the server, which carries the input image and text data; ② The server responds to the image processing request, and after obtaining the input image and text data, it can encode the text data to obtain text embedding features; and, according to multiple data dimensions, perform image feature extraction on the preset object in the input image to obtain the identity embedding features of the preset object in each data dimension; ③ The server adopts an interleaved adjustment mechanism and calls the text generation image model to perform fusion recognition processing on the identity embedding features and text embedding features of the preset object in each data dimension to generate an output image, that is, generate a personalized image that matches the preset object; ④ The server sends the personalized image to the terminal device. In the above-mentioned personalized image generation process, on the one hand, based on the staggered adjustment mechanism, the problem that user identity features are the dominant factor and text embedding features lose control of personalized images can be solved. This mechanism allows different conditions to be added independently without conflict, thereby balancing the differences between global identity embedding features and local identity embedding features; on the other hand, the present application obtains rich texture information from the reference spatial features, and can introduce a new parallel mutual attention layer on the basis of the self-attention layer in the text-generated image model to embed local spatial information (i.e., local identity embedding features), thereby extracting more texture details in the local identity embedding features, providing a more accurate prerequisite for the personalized image generation process, so that personalized images can be generated more accurately.

[0126] The above details the method of the embodiment of the present application. In order to facilitate the better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below. Next, combined with the image processing scheme provided by the above embodiment of the present application, the relevant devices of the embodiment of the present application are introduced accordingly.

[0127] Please refer to Figure 9, which is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. As shown in Figure 9, the image processing device 900 can be applied to the computer device (such as a terminal device or a server) mentioned in the aforementioned embodiment. Specifically, the image processing device 900 can be a computer program (including program code) running in a computer device, for example, the image processing device 900 is an application software; the image processing device 900 can be used to execute the corresponding steps in the image processing method provided in an embodiment of the present application. In specific implementation, the image processing device 900 can specifically include:

[0128] An acquisition unit 901 is configured to acquire an input image containing a preset object and acquire text data;

[0129] Processing unit 902, configured to encode text data to obtain text embedding features;

[0130] The processing unit 902 is further configured to perform image feature extraction processing on the input image according to a plurality of predefined data dimensions to obtain identity embedding features of the preset object in each data dimension;

[0131] The processing unit 902 is also used to call the text-generated image model to perform fusion recognition processing on the text embedding features of the preset object and the identity embedding features of each data dimension to generate an output image, wherein the text-generated image model includes multiple cross-attention layers, and the text embedding features and the identity embedding features of each data dimension are alternately input into the cross-attention layers; the output image includes image features related to the preset object in the input image described by the text data.

[0132] In a possible implementation, the data dimension includes a global dimension and a local dimension; the identity embedding feature of the preset object in the global dimension is represented as a global identity embedding feature, and the identity embedding feature of the preset object in the local dimension is represented as a local identity embedding feature;

[0133] When the processing unit 902 performs image feature extraction processing on the preset object in the input image according to multiple predefined data dimensions to obtain the identity embedding features of the preset object in each data dimension, it is configured to perform the following operations:

[0134] A global identity encoder is called to perform identity recognition processing on the preset object in the input image according to the global dimension to obtain the global identity embedding feature of the preset object; and a local texture encoder is called to extract texture features of the preset object in the input image according to the local dimension to obtain the local identity embedding feature of the preset object.

[0135] In one possible implementation, the processing unit 902 is further configured to perform the following operations before calling the global identity encoder to perform identity recognition processing on the preset object in the input image according to the global dimension to obtain the global identity embedding feature of the preset object:

[0136] Perform detection processing on a preset object in an input image to obtain a detection result; perform image segmentation processing on the input image according to the detection result to obtain a regional image containing the preset object; obtain the spatial resolution of the global identity encoder, and perform image adjustment processing on the regional image according to the spatial resolution to obtain a regional adjusted image; wherein the image adjustment processing includes: adjusting the resolution; in the process of performing image adjustment processing on the regional image, also including any one or both of adjusting the clarity of the regional image and adjusting the image size.

[0137] In one possible implementation, the global identity encoder includes: a first image sub-encoder and a second image sub-encoder; when the processing unit 902 calls the global identity encoder to perform identity recognition processing on a preset object in the input image according to a global dimension to obtain a global identity embedding feature of the preset object, it is configured to perform the following operations:

[0138] Calling the first image sub-encoder to perform image encoding processing on the preset object contained in the region-adjusted image to obtain a first image feature of the preset object, where the first image feature is used to reflect the attribute feature of the preset object; calling the second image sub-encoder to perform image encoding processing on the preset object contained in the region-adjusted image to obtain a second image feature of the preset object, where the second image feature is used to reflect the classification feature of the preset object; based on the first image feature and the second image feature, obtaining a global identity embedding feature of the preset object.

[0139] In a possible implementation, when obtaining the global identity embedding feature of the preset object based on the first image feature and the second image feature, the processing unit 902 is configured to perform any one of the following:

[0140] Performing an average operation on the first image feature and the second image feature, and determining the image feature obtained after the average operation as the global identity embedding feature of the preset object;

[0141] Performing a weighted operation on the first image feature and the second image feature, and determining the image feature obtained after the weighted operation as a global identity embedding feature of the preset object;

[0142] Feature alignment processing is performed on the first image feature and the second image feature, and the first image feature and the second image feature after feature alignment are combined into a global identity embedding feature of a preset object.

[0143] In one possible implementation, the identity embedding features include: a global identity embedding feature and a local identity embedding feature; the processing unit 902, when calling the text-to-image model to perform fusion recognition processing on the text embedding features of the preset object and the identity embedding features of each data dimension to generate an output image, is configured to perform the following operations:

[0144] The text-generated image model is called to perform interlaced adjustment processing on the global identity embedding features and text embedding features of the preset object to obtain a first feature result; the text-generated image model is called to perform feature enhancement processing on the local identity embedding features of the preset object to obtain a second feature result; the text-generated image model is called to decode and identify the first feature result and the second feature result to generate an output image.

[0145] In one possible implementation, the text-generated image model includes k cross-attention layers, where any cross-attention layer is represented as the jth cross-attention layer, where k and j are both positive integers, and j≤k; when the processing unit 902 calls the text-generated image model to perform interleaved adjustment processing on the global identity embedding feature and the text embedding feature of the preset object to obtain a first feature result, it is configured to perform the following operations:

[0146] The global identity embedding feature is input into the jth cross attention layer, and the text embedding feature is input into the j+1th cross attention layer; until the k cross attention layers are all input with the global identity embedding feature or the text embedding feature, the global identity embedding feature and the text embedding feature are interlaced and adjusted based on the k cross attention layers to obtain the first feature result.

[0147] In one possible implementation, any cross-attention layer having a global identity embedding feature as input is denoted as Si, and any cross-attention layer having a text embedding feature as input is denoted as Sj; when the processing unit 902 performs interleaved adjustment processing on the global identity embedding feature and the text embedding feature based on k cross-attention layers and obtains a first feature result, it is configured to perform the following operations:

[0148] Based on each cross-attention layer Si in the k cross-attention layers, attention extraction processing is performed on the global identity embedding feature to obtain the attention feature of the global identity embedding feature; based on each cross-attention layer Sj in the k cross-attention layers, attention extraction processing is performed on the text embedding feature to obtain the attention feature of the text embedding feature; the attention feature of the global identity embedding feature and the attention feature of the text embedding feature are fused to obtain the first feature result.

[0149] In one possible implementation, the text-to-image model includes a mutual attention layer and a self-attention layer. When the processing unit 902 calls the text-to-image model to perform feature enhancement processing on the local identity embedding feature of the preset object to obtain a second feature result, it is configured to perform the following operations:

[0150] Obtain a background embedding feature of the input image, which is obtained after image recognition of the background image other than the preset object in the input image; call the self-attention layer to perform recognition processing on the background embedding feature to obtain a self-attention recognition result; and call the mutual attention layer to perform recognition processing on the local identity embedding feature of the preset object to obtain a mutual attention recognition result; perform feature enhancement processing on the self-attention recognition result and the mutual attention recognition result to obtain a second feature result.

[0151] In one possible implementation, the local identity embedding feature includes multiple spatial embedding features; any two spatial embedding features are represented as: Yk and Yv; the processing unit 902, when calling the mutual attention layer to perform recognition processing on the local identity embedding feature of the preset object and obtaining the mutual attention recognition result, is used to perform the following operations:

[0152] The spatial embedding feature Yk is used as the key of the mutual attention layer, and the spatial embedding feature Yv is used as the value of the mutual attention layer; the mutual attention weight of the mutual attention layer is obtained; the spatial embedding features included in the local identity embedding feature are fused according to the mutual attention weight to obtain the mutual attention processing result.

[0153] In a possible implementation, when the processing unit 902 encodes the text data to obtain the text embedding feature, it is configured to perform the following operations:

[0154] The text data is segmented to obtain multiple text words, and multiple keywords are extracted from the multiple text words in the text data; the text encoder is called to perform feature extraction on each extracted keyword to obtain a word vector for each keyword; and text embedding features are generated based on each extracted word vector.

[0155] In the embodiments of the present application, the specific implementation of the operations performed by each unit of the device and the corresponding effects that can be produced can be referred to the relevant descriptions of the aforementioned embodiments and will not be repeated here.

[0156] Please refer to Figure 10, which is a structural diagram of a computer device provided in an embodiment of the present application. The computer device 1000 is used to execute the steps performed by the terminal device or server in the aforementioned method embodiment. The computer device 1000 includes: one or more processors 1001; one or more input devices 1002, one or more output devices 1003 and a memory 1004. The above-mentioned processors 1001, input devices 1002, output devices 1003 and memory 1004 are connected via a bus 1005. Specifically, the memory 1004 is used to store a computer program, and the computer program includes program instructions. The processor 1001 is used to call the program instructions stored in the memory 1004 to perform the following operations:

[0157] An input image containing a preset object and text data are obtained; the text data is encoded to obtain text embedding features; image feature extraction is performed on the input image according to a plurality of predefined data dimensions to obtain identity embedding features of the preset object in each data dimension; a text generation image model is called to perform fusion recognition processing on the text embedding features of the preset object and the identity embedding features of each data dimension to generate an output image, wherein the text generation image model includes a plurality of cross-attention layers, and the text embedding features and the identity embedding features of each data dimension are alternately input into the cross-attention layers; the output image includes image features related to the preset object in the input image described by the text data.

[0158] In one possible implementation, the data dimension includes a global dimension and a local dimension; the identity embedding feature of the preset object in the global dimension is represented as a global identity embedding feature, and the identity embedding feature of the preset object in the local dimension is represented as a local identity embedding feature; when the processor 1001 performs image feature extraction processing on the preset object in the input image according to multiple predefined data dimensions and obtains the identity embedding feature of the preset object in each data dimension, it is configured to perform the following operations:

[0159] A global identity encoder is called to perform identity recognition processing on the preset object in the input image according to the global dimension to obtain the global identity embedding feature of the preset object; and a local texture encoder is called to extract texture features of the preset object in the input image according to the local dimension to obtain the local identity embedding feature of the preset object.

[0160] In one possible implementation, before calling the global identity encoder to perform identity recognition processing on the preset object in the input image according to the global dimension to obtain the global identity embedding feature of the preset object, the processor 1001 is further configured to perform the following operations:

[0161] Perform detection processing on a preset object in an input image to obtain a detection result; perform image segmentation processing on the input image according to the detection result to obtain a regional image containing the preset object; obtain the spatial resolution of the global identity encoder, and perform image adjustment processing on the regional image according to the spatial resolution to obtain a regional adjusted image; wherein the image adjustment processing includes: adjusting the resolution; in the process of performing image adjustment processing on the regional image, also including any one or both of adjusting the clarity of the regional image and adjusting the image size.

[0162] In one possible implementation, the global identity encoder includes: a first image sub-encoder and a second image sub-encoder; when the processor 1001 calls the global identity encoder to perform identity recognition processing on a preset object in an input image according to a global dimension to obtain a global identity embedding feature of the preset object, it is configured to perform the following operations:

[0163] Calling a first image sub-encoder to perform image encoding processing on a preset object contained in the region-adjusted image to obtain a first image feature of the preset object, where the first image feature is used to reflect the attribute feature of the preset object; calling a second image sub-encoder to perform image encoding processing on the preset object contained in the region-adjusted image to obtain a second image feature of the preset object, where the second image feature is used to reflect the classification feature of the preset object; and obtaining a global identity embedding feature of the preset object based on the first image feature and the second image feature.

[0164] In a possible implementation, when obtaining the global identity embedding feature of the preset object based on the first image feature and the second image feature, the processor 1001 is configured to perform any one of the following:

[0165] Performing an average operation on the first image feature and the second image feature, and determining the image feature obtained after the average operation as the global identity embedding feature of the preset object;

[0166] Performing a weighted operation on the first image feature and the second image feature, and determining the image feature obtained after the weighted operation as a global identity embedding feature of the preset object;

[0167] Feature alignment processing is performed on the first image feature and the second image feature, and the first image feature and the second image feature after feature alignment are combined into a global identity embedding feature of a preset object.

[0168] In one possible implementation, the identity embedding features include: a global identity embedding feature and a local identity embedding feature; when the processor 1001 calls the text-to-image model to perform fusion recognition processing on the text embedding features of the preset object and the identity embedding features of each data dimension to generate an output image, it is configured to perform the following operations:

[0169] The text-generated image model is called to perform interlaced adjustment processing on the global identity embedding features and text embedding features of the preset object to obtain a first feature result; the text-generated image model is called to perform feature enhancement processing on the local identity embedding features of the preset object to obtain a second feature result; the text-generated image model is called to decode and identify the first feature result and the second feature result to generate an output image.

[0170] In one possible implementation, the text-generated image model includes k cross-attention layers, where any cross-attention layer is represented as the jth cross-attention layer, where k and j are both positive integers, and j≤k; the processor 1001 adopts an interleaved adjustment mechanism and calls the text-generated image model to perform interleaved adjustment processing on the global identity embedding feature and the text embedding feature of the preset object, and when a first feature result is obtained, is configured to perform the following operations:

[0171] The global identity embedding feature is input into the jth cross attention layer, and the text embedding feature is input into the j+1th cross attention layer; until the k cross attention layers are all input with the global identity embedding feature or the text embedding feature, the global identity embedding feature and the text embedding feature are interlaced and adjusted based on the k cross attention layers to obtain the first feature result.

[0172] In one possible implementation, any cross-attention layer having a global identity embedding feature as input is denoted as Si, and any cross-attention layer having a text embedding feature as input is denoted as Sj; when the processor 1001 performs interleaved adjustment processing on the global identity embedding feature and the text embedding feature based on k cross-attention layers and obtains a first feature result, it is configured to perform the following operations:

[0173] Based on each cross-attention layer Si in the k cross-attention layers, attention extraction processing is performed on the global identity embedding feature to obtain the attention feature of the global identity embedding feature; based on each cross-attention layer Sj in the k cross-attention layers, attention extraction processing is performed on the text embedding feature to obtain the attention feature of the text embedding feature; the attention feature of the global identity embedding feature and the attention feature of the text embedding feature are fused to obtain the first feature result.

[0174] In one possible implementation, the text-to-image model includes a mutual attention layer and a self-attention layer. When the processor 1001 calls the text-to-image model to perform feature enhancement processing on the local identity embedding feature of the preset object to obtain a second feature result, it is configured to perform the following operations:

[0175] Obtain a background embedding feature of the input image, which is obtained after image recognition of the background image other than the preset object in the input image; call the self-attention layer to perform recognition processing on the background embedding feature to obtain a self-attention recognition result; and call the mutual attention layer to perform recognition processing on the local identity embedding feature of the preset object to obtain a mutual attention recognition result; perform feature enhancement processing on the self-attention recognition result and the mutual attention recognition result to obtain a second feature result.

[0176] In one possible implementation, the local identity embedding feature includes multiple spatial embedding features; any two spatial embedding features are represented as: Yk and Yv; when the processor 1001 calls the mutual attention layer to identify the local identity embedding feature of the preset object and obtains the mutual attention recognition result, it is used to perform the following operations:

[0177] The spatial embedding feature Yk is used as the key of the mutual attention layer, and the spatial embedding feature Yv is used as the value of the mutual attention layer; the mutual attention weight of the mutual attention layer is obtained; the spatial embedding features included in the local identity embedding feature are fused according to the mutual attention weight to obtain the mutual attention processing result.

[0178] In a possible implementation, when the processing unit 902 encodes the text data to obtain the text embedding feature, it is configured to perform the following operations:

[0179] The text data is segmented to obtain multiple text words, and multiple keywords are extracted from the multiple text words in the text data; a text encoder is called to perform feature extraction on each extracted keyword to obtain a word vector for each keyword; and text embedding features are generated based on each extracted word vector.

[0180] In the embodiments of the present application, the specific implementation of each operation performed by the processor of the computer device and the corresponding effects that can be produced can be referred to the relevant descriptions of the aforementioned embodiments and will not be repeated here.

[0181] In addition, it should be noted here that: the embodiment of the present application also provides a computer storage medium, and a computer program is stored in the computer storage medium, and the computer program includes program instructions. When the processor executes the above program instructions, it can execute the method in the corresponding embodiment above, so it will not be described in detail here. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located in one place, or, executed on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0182] According to one aspect of the present application, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, so that the computer device can perform the methods described in the corresponding embodiments above. Therefore, these methods will not be described in detail here.

[0183] Those skilled in the art will appreciate that the units and algorithmic steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0184] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data processing device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0185] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, characterized in that: include: Get an input image containing a preset object and obtain text data; Encoding the text data to obtain text embedding features; Performing image feature extraction processing on the input image according to a plurality of predefined data dimensions to obtain identity embedding features of the preset object in each data dimension; The text-generated image model is called to perform fusion recognition processing on the text embedding features of the preset object and the identity embedding features of each data dimension to generate an output image, wherein the text-generated image model includes a plurality of cross-attention layers, and the text embedding features and the identity embedding features of each data dimension are alternately input into the cross-attention layers; the output image includes image features related to the preset object in the input image described by the text data.

2. The method according to claim 1, characterized in that The data dimension includes a global dimension and a local dimension; the identity embedding feature of the preset object in the global dimension is represented as a global identity embedding feature, and the identity embedding feature of the preset object in the local dimension is represented as a local identity embedding feature; The performing image feature extraction processing on the input image according to a plurality of predefined data dimensions to obtain identity embedding features of the preset object in each data dimension includes: Calling a global identity encoder to perform identity recognition processing on a preset object in the input image according to a global dimension to obtain a global identity embedding feature of the preset object; A local texture encoder is called to extract texture features of a preset object in the input image according to a local dimension to obtain a local identity embedding feature of the preset object.

3. The method according to claim 1 or 2, characterized in that Before calling the global identity encoder to perform identity recognition processing on the preset object in the input image according to the global dimension to obtain the global identity embedding feature of the preset object, the method further includes: Performing detection processing on a preset object in the input image to obtain a detection result; Performing image segmentation processing on the input image according to the detection result to obtain a regional image containing a preset object; Acquire the spatial resolution of the global identity encoder, and perform image adjustment processing on the regional image according to the spatial resolution to obtain a regional adjusted image; wherein the image adjustment processing includes: adjusting the resolution; The process of performing image adjustment processing on the regional image also includes either or both of adjusting the clarity and the image size of the regional image.

4. The method according to any one of claims 1 to 3, characterized in that: The global identity encoder comprises: a first image sub-encoder and a second image sub-encoder; calling the global identity encoder to perform identity recognition processing on a preset object in the input image according to a global dimension to obtain a global identity embedding feature of the preset object comprises: Calling a first image sub-encoder to perform image encoding processing on a preset object included in the region adjustment image to obtain a first image feature of the preset object, where the first image feature is used to reflect an attribute feature of the preset object; calling a second image sub-encoder to perform image encoding processing on a preset object included in the region-adjusted image to obtain a second image feature of the preset object, where the second image feature is used to reflect a classification feature of the preset object; Based on the first image feature and the second image feature, a global identity embedding feature of the preset object is obtained.

5. The method according to claim 4, characterized in that obtaining a global identity embedding feature of the preset object based on the first image feature and the second image feature, Includes any of the following: Performing an average operation on the first image feature and the second image feature, and determining the image feature obtained after the average operation as the global identity embedding feature of the preset object; Performing a weighted operation on the first image feature and the second image feature, and determining the image feature obtained after the weighted operation as the global identity embedding feature of the preset object; The first image feature and the second image feature are subjected to feature alignment processing, and the first image feature and the second image feature after feature alignment are combined into a global identity embedding feature of the preset object.

6. The method according to any one of claims 1 to 5, characterized in that: The identity embedding features include: global identity embedding features and local identity embedding features; the calling text generation image model performs fusion recognition processing on the text embedding features of the preset object and the identity embedding features of each data dimension to generate an output image, including: Calling a text-generated image model to perform interlaced adjustment processing on the global identity embedding feature of the preset object and the text embedding feature to obtain a first feature result; Calling the text-to-image model to perform feature enhancement processing on the local identity embedding feature of the preset object to obtain a second feature result; The text-to-image model is called to perform decoding and recognition processing on the first feature result and the second feature result to generate an output image.

7. The method according to claim 6, characterized in that The text generation image model includes k cross-attention layers, any cross-attention layer is represented as the jth cross-attention layer, k and j are both positive integers, and j≤k; The calling text generation image model performs interlaced adjustment processing on the global identity embedding feature of the preset object and the text embedding feature to obtain a first feature result, including: Input the global identity embedding feature into the jth crisscross attention layer, and input the text embedding feature into the j+1th crisscross attention layer; Until the k cross-attention layers are all input with the global identity embedding feature or the text embedding feature, the global identity embedding feature and the text embedding feature are respectively subjected to interlaced adjustment processing based on the k cross-attention layers to obtain a first feature result.

8. The method according to claim 7, characterized in that Denote any cross attention layer input with the global identity embedding feature as Si, and denote any cross attention layer input with the text embedding feature as Sj; and perform interlaced adjustment processing on the global identity embedding feature and the text embedding feature based on the k cross attention layers, respectively, to obtain a first feature result, including: Performing attention extraction processing on the global identity embedding feature based on each cross attention layer Si in the k cross attention layers to obtain an attention feature of the global identity embedding feature; Performing attention extraction processing on the text embedding feature based on each cross attention layer Sj in the k cross attention layers to obtain an attention feature of the text embedding feature; The attention feature of the global identity embedding feature and the attention feature of the text embedding feature are fused to obtain a first feature result.

9. The method according to any one of claims 1 to 8, characterized in that: The text-to-image model includes: a mutual attention layer and a self-attention layer; The calling of the text generation image model performs feature enhancement processing on the local identity embedding feature of the preset object to obtain a second feature result, including: Acquire a background embedding feature of the input image, where the background embedding feature is obtained after performing image recognition on a background image other than the preset object in the input image; Calling the self-attention layer to perform recognition processing on the background embedding feature to obtain a self-attention recognition result; and, Calling the mutual attention layer to perform recognition processing on the local identity embedding features of the preset object to obtain a mutual attention recognition result; Feature enhancement processing is performed on the self-attention recognition result and the mutual attention recognition result to obtain a second feature result.

10. The method according to any one of claims 1 to 9, characterized in that: The local identity embedding feature includes a plurality of spatial embedding features; Any two spatial embedding features are represented as: Yk and Yv; The calling of the mutual attention layer to perform recognition processing on the local identity embedding feature of the preset object to obtain a mutual attention recognition result includes: Use the spatial embedding feature Yk as the key of the mutual attention layer, and use the spatial embedding feature Yv as the value of the mutual attention layer; Obtaining the mutual attention weight of the mutual attention layer; The various spatial embedding features included in the local identity embedding feature are fused according to the mutual attention weight to obtain a mutual attention processing result.

11. The method according to any one of claims 1 to 10, characterized in that: The encoding process of the text data to obtain text embedding features includes: Performing word segmentation processing on the text data to obtain a plurality of text words, and extracting a plurality of keywords from the plurality of text words in the text data; Call the text encoder to perform feature extraction on each extracted keyword to obtain the word vector of each keyword; Generate text embedding features based on the extracted word vectors.

12. An image processing device, characterized in that: include: An acquisition unit, used to acquire an input image containing a preset object and acquire text data; A processing unit, used for encoding the text data to obtain text embedding features; The processing unit is further used to perform image feature extraction processing on the input image according to multiple predefined data dimensions to obtain identity embedding features of the preset object in each data dimension; The processing unit is also used to call the text-generated image model to perform fusion recognition processing on the text embedding features of the preset object and the identity embedding features of each data dimension to generate an output image, wherein the text-generated image model includes a plurality of cross-attention layers, and the text embedding features and the identity embedding features of each data dimension are alternately input into the cross-attention layers; the output image includes image features related to the preset object in the input image described by the text data.

13. A computer device, characterized in that: include: storage devices and processors; a memory storing one or more computer programs; A processor, configured to load the one or more computer programs to implement the image processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the image processing method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program, which is suitable for being loaded by a processor and executing the image processing method according to any one of claims 1-11.

Images