Visual task processing method, visual processing model training method, visual processing model-based information processing method and task platform

Abstract

The embodiment of the invention provides a visual task processing method, a visual processing model training method, an information processing method based on a visual processing model and a task platform, and the visual task processing method comprises the steps that task data of a target visual task are acquired, and the task data comprise at least one visual object; extracting object embedding features of the at least one visual object, and determining spatial distribution information of the at least one visual object according to the object embedding features of the at least one visual object; and according to the spatial distribution information of the at least one visual object, distributing the at least one visual object to a corresponding object region by using a target visual processing model, and obtaining a target processing result of the target visual task. And allocating the at least one visual object to the corresponding object area by using the target visual processing model, obtaining a target processing result of the target visual task, allocating and managing the at least one visual object, and realizing customized generation of a single-visual object and a multi-visual object.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of deep learning technology, and in particular to visual task processing, training of visual processing models, and information processing methods and task platforms based on visual processing models. Background Art

[0002] With the rapid development of computer technology and artificial intelligence technology, deep learning models have shown great potential in visual task processing and can perform different downstream visual tasks. At present, when processing visual tasks, there are many restrictions on visual objects and text prompts, and the flexibility and accuracy of visual task processing are poor. Therefore, there is an urgent need for a visual task processing method that can improve the flexibility and accuracy of task processing. Summary of the invention

[0003] In view of this, an embodiment of this specification provides a visual task processing method. One or more embodiments of this specification also relate to a training method of a visual processing model, an information processing method based on a visual processing model, a task platform, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a visual task processing method is provided, including:

[0005] Acquire task data of a target visual task, wherein the task data includes at least one visual object;

[0006] extracting an object embedding feature of the at least one visual object, and determining spatial distribution information of the at least one visual object according to the object embedding feature of the at least one visual object;

[0007] According to the spatial distribution information of the at least one visual object, the at least one visual object is allocated to a corresponding object area using a target visual processing model to obtain a target processing result of the target visual task.

[0008] According to a second aspect of an embodiment of this specification, a method for training a visual processing model is provided, comprising:

[0009] Acquire training data for a training task, wherein the training data includes at least one sample object and a corresponding result label;

[0010] extracting a sample object embedding feature of the at least one sample object, and determining sample space distribution information of the at least one sample object according to the sample object embedding feature of the at least one sample object;

[0011] The initial visual processing model is trained according to the sample space distribution information of the at least one sample object in combination with the result label to obtain a trained target visual processing model.

[0012] According to a third aspect of an embodiment of this specification, there is provided an information processing method based on a visual processing model, which is applied to a task platform and includes:

[0013] Receiving a model request sent by a terminal device, wherein the model request includes a scene identifier of a target scene, scene input data of the target scene, and at least one of a model specification parameter;

[0014] Based on the model request, a corresponding target visual processing model is determined from at least one visual processing model, wherein the at least one visual processing model is trained based on the visual processing model training method as described above.

[0015] According to a fourth aspect of the embodiments of this specification, there is provided a task platform, including a request interface and a response unit;

[0016] The request interface is used to receive a model request sent by a terminal device, wherein the model request includes a scene identifier of a target scene, scene input data of the target scene, and at least one of a model specification parameter;

[0017] The response unit is used to determine a corresponding target visual processing model from at least one visual processing model based on the model request, wherein the at least one visual processing model is trained based on the visual processing model training method as described above.

[0018] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0019] Memory and processor;

[0020] Among them, the memory is used to store computer programs / instructions, and the processor is used to execute computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the above-mentioned visual task processing method, the training method of the visual processing model, or the information processing method based on the visual processing model are implemented.

[0021] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual task processing method, visual processing model training method, or information processing method based on a visual processing model.

[0022] According to the seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual task processing method, visual processing model training method, or information processing method based on a visual processing model.

[0023] An embodiment of the present specification realizes the extraction of object embedding features of at least one visual object, and based on the object embedding features of at least one visual object, determines the spatial distribution information of at least one visual object, uses a target visual processing model to assign at least one visual object to a corresponding object area, obtains a target processing result of a target visual task, adaptively combines and distributes at least one visual object to a corresponding object area, assigns and manages at least one visual object, and realizes customized generation of single visual objects and multiple visual objects without restricting text prompts and layout of the target visual processing model, without requiring manual intervention, without affecting the integrity of the visual objects, the diversity of the target processing results and the flexibility of the text prompts, thereby improving the accuracy of visual task processing, and being compatible with most target visual processing models, achieving good scalability and general compatibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is an application architecture diagram of a visual task provided by an embodiment of this specification;

[0025] Figure 2 is a flow chart of a visual task processing method provided by one embodiment of this specification;

[0026] Figure 3 is a schematic diagram of a processing process of a visual task processing method provided by an embodiment of this specification;

[0027] Figure 4 is a schematic diagram of a visual task processing interface provided by an embodiment of this specification;

[0028] Figure 5 is a schematic diagram of another visual task processing interface provided by an embodiment of this specification;

[0029] Figure 6 is a flow chart of a method for training a visual processing model provided by an embodiment of this specification;

[0030] Figure 7 It is a schematic diagram of a processing process of a training method of a visual processing model provided by an embodiment of this specification;

[0031] Figure 8 is a flow chart of an information processing method based on a visual processing model provided by an embodiment of this specification;

[0032] Fig. 9 It is a structural diagram of a task platform provided by an embodiment of this specification;

[0033] Fig.10 is a structural schematic diagram of a visual task processing device provided by an embodiment of this specification;

[0034] Fig.11 It is a structural schematic diagram of a training device for a visual processing model provided by one embodiment of this specification;

[0035] Fig.12 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0036] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0037] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0038] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0039] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0040] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model), a multi-modal pre-training model, etc.

[0041] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC, Image Caption), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0042] First, the terms involved in one or more embodiments of this specification are explained.

[0043] Face recognition model: usually called face representation learning network or face feature extraction network, it is a core component in face recognition technology. It plays a vital role in face recognition system. Its main function is to extract high-dimensional and discriminative feature vectors from input face images. These feature vectors can effectively represent the uniqueness of each face, so that different individuals can be accurately distinguished.

[0044] Multilayer Perceptron (MLP): It is a basic type of artificial neural network (ANN) and one of the earliest widely studied and applied neural network models. MLP is a feedforward neural network, which means that information flows unidirectionally from the input layer to the output layer between neurons, without loops or feedback connections.

[0045] CLIP (Contrastive Language-Image Pre-training): is an artificial intelligence model proposed by OpenAI, which achieves cross-modal understanding and alignment between images and text by pre-training on large-scale Internet data. The core of CLIP lies in its innovative training method, which uses contrastive learning, with the goal of learning how to pair images with the text describing them without direct manual labels.

[0046] FFN (Feedforward Neural Network): It is the most basic type of artificial neural network, in which information flows unidirectionally along the network, from the input layer through a series of intermediate hidden layers (if any), and finally reaches the output layer. The transmission of information during this period is unidirectional and non-cyclic. In a more specific context, especially when mentioned in the Transformer architecture, FFN refers to Position-wise Feed-Forward Networks. This is one of the two main components in the Transformer model, the other being the Self-Attention mechanism. FFN appears in every encoder and decoder layer of the Transformer. The FFN structure in the Transformer usually contains two fully connected layers (also called densely connected layers or linear layers) and a non-linear activation function. Common activation functions include ReLU (Rectified Linear Unit) or GELU (Gaussian Error Linear Unit).

[0047] Diffusion Model: Also known as diffusion probability model in machine learning, it is a class of latent variable models. The goal of diffusion models is to learn the latent structure of a dataset by modeling how data points diffuse in the latent space. In computer vision, the transformation from noise to image latent space can be learned by modeling how data diffuses in the latent space, which means that a neural network is trained to denoise an image blurred by Gaussian noise by learning to reverse the diffusion process. Diffusion models can be applied to a variety of tasks, including image denoising, picture-in-picture, super-resolution, and image generation. For example, an image generation model will start with a random noisy image, and then after training on natural images to reverse the diffusion process, the model will be able to generate new natural images.

[0048] It should be noted that with the rapid development of artificial intelligence and computer vision technology, personalized image generation has gradually become a hot research field. The personalized generation framework responds to users' growing demand for personalized and customized content, and can provide users with a more creative and personalized visual experience.

[0049] At present, the character posture, face position, etc. can be determined before generating the image to realize the customized generation of text to single-person and multi-person images without the need for other human intervention. This implementation method needs to first identify the image to determine the character posture, face position, etc., or artificially limit the character posture, face position, etc., which requires pre-image processing or artificial definition. This implementation method has poor identity fidelity and cannot accurately customize the user's face; secondly, due to the limitation of character posture and face position, the model may pay too much attention to the character posture and face position, have a certain overfitting of the face, and have poor editability; thirdly, there are restrictions on the text prompt words describing the character, for example, only singular phrases are supported to describe the character subject.

[0050] In another possible implementation method, there are other models that can realize the ability to generate personalized characters, but most models need to fine-tune the training model for each new character, which consumes a lot of computing resources. Most models have the ability to generate customized text to a single character, but cannot generate text to multiple characters, or require artificially fixing the character's posture and face position to achieve multiple character generation.

[0051] Therefore, in one embodiment of this specification, based on the diffusion model, a personalized image generation framework is provided, which does not require fine-tuning the training model for each new character, is fast and reduces resource consumption, combines the needs of single-character and multi-character generation, and achieves good scalability and universal compatibility through high fidelity, wide face editability, free-form text input and other features, meets the user's needs for personalized generated content, and provides users with a more free and personalized creative space. This solution brings new application prospects to the fields of character style transfer, personalized portrait generation, character stylization, etc., and is widely used in virtual image creation, personalized advertising promotion, personalized creative expression, etc. It can more accurately display the user's image and create more creative and personalized visual content.

[0052] To solve the above technical problems, a visual task processing method is provided in this specification. One or more embodiments of this specification also involve a training method of a visual processing model, an information processing method based on a visual processing model, a task platform, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0053] Considering the huge number of model parameters of large models and the limited computing resources of mobile terminals, the visual task processing method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 This is an application architecture diagram of a visual task provided by an embodiment of this specification. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, vehicle-mounted devices, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiments of this specification.

[0054] In an embodiment of the present specification, a system composed of a client device and a server can perform the following steps: the client device executes to receive the task data of the target visual task input by the user and uploads it to the server. The server executes to obtain the task data of the target visual task, wherein the task data includes at least one visual object; extracts the object embedding features of at least one visual object, and determines the spatial distribution information of at least one visual object based on the object embedding features of at least one visual object; according to the spatial distribution information of at least one visual object, uses the target visual processing model to allocate at least one visual object to the corresponding object area, and obtains the target processing result of the target visual task. In this way, at least one visual object is adaptively combined and distributed to the corresponding object area, and at least one visual object is allocated and managed to realize the customized generation of single visual objects and multiple visual objects. It should be noted that when the operating resources of the client device can meet the deployment and operation conditions of the large model, the embodiment of the present application can be carried out in the client device.

[0055] See also Figure 2 , Figure 2 A flowchart of a visual task processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps 202-206.

[0056] Step 202: Acquire task data of a target visual task, wherein the task data includes at least one visual object.

[0057] The embodiments of this specification are applied to applications, websites or mini-programs with visual task processing functions. On applications, websites or mini-programs, visual task processing functions are implemented. For example, a website that deploys a target visual processing model implements task processing functions for visual tasks such as image generation, virtual image creation, advertisement generation, style transfer, etc. For another example, a third-party application calls the deployed target visual processing model through an application programming interface (Application Programming Interface, referred to as API) to implement the processing function of the corresponding visual task.

[0058] The target visual task is a task to be processed, and the target visual task is used to instruct visual processing based on task data, such as a text image task, a virtual image creation task, an advertisement generation task, a style transfer task, etc. The task data may include at least one visual object, which is an object required to be displayed in the target processing result of the target visual task, such as a person, a face, an animal, a virtual image, etc.

[0059] In an optional implementation of this embodiment, task data of the target visual task may be obtained based on the front-end input, that is, the task data of the target visual task may be obtained, including:

[0060] A target visual task is generated in response to at least one visual object and text prompt information input by a front end.

[0061] Among them, the text prompt information is used to indicate the processing method of at least one visual object to obtain the corresponding target processing result, that is, the text prompt information can describe the target processing result that needs to be generated, thereby guiding the target visual processing model to process at least one visual object according to the text prompt information to obtain the corresponding target processing result.

[0062] For example, a user may upload an image of three faces, and a text prompt message "three little girls posing together on a swing, elegantly dressed". Subsequently, the target visual processing model may generate a target image, which is an image of "three little girls posing together on a swing, elegantly dressed", and the "three little girls" in the target image are the three faces uploaded by the user.

[0063] In the embodiments of the present specification, a target visual task can be generated in response to at least one visual object and text prompt information input by the front end. The processing method of at least one visual object is described by the text prompt information. The user can independently select the visual object to be processed and the corresponding processing method. A rich interaction method is provided, which can realize different visual task processing and meet different task requirements. The operation threshold is low, and the user experience is improved.

[0064] Of course, in actual implementation, in addition to the user uploading at least one visual object and entering text prompt information, the user can also input information to generate "at least one visual object, and text prompt information". For example, the user inputs "a smiling woman with loose hair, wearing a necklace, jogging on the lawn", and a face that meets the requirements of "a smiling woman with loose hair, wearing a necklace" can be automatically generated as a visual object, and "a smiling woman with loose hair, wearing a necklace, jogging on the lawn" can be used as a text prompt information. Subsequently, the target visual processing model can generate a target image, which is an image of "a smiling woman with loose hair, wearing a necklace, jogging on the lawn", and the face in the target image is a visual object generated based on the user input information, thereby realizing the customized visual task of the Wensheng map.

[0065] In another optional implementation, the target visual task may further include a reference image, which may be selected by a user. Subsequently, the target visual processing model may be used to synthesize at least one visual object into the reference image according to the prompt of the text prompt information to obtain the target image as the target processing result.

[0066] Step 204: extracting object embedding features of at least one visual object, and determining spatial distribution information of at least one visual object according to the object embedding features of the at least one visual object.

[0067] Among them, object embedding features refer to the features obtained by extracting image features from visual objects.

[0068] In actual implementation, a model, algorithm, software tool or programming technique may be used to process at least one visual object to obtain an object embedding feature of at least one visual object. In addition, a model, algorithm, software tool or programming technique may be used to process the object embedding feature of at least one visual object to determine the spatial distribution information of at least one visual object. The spatial distribution information of at least one visual object is used to indicate the object area where at least one visual object is located, and is used to indicate that at least one visual object is adaptively combined and distributed to the corresponding area in the target processing result to generate the target processing result of the target visual task.

[0069] In a possible implementation, at least one visual object may be input into an object embedding model, and object embedding features of each visual object may be extracted through the object embedding model, wherein the object embedding model may be any model capable of performing image encoding on the input visual object to extract object embedding features. Furthermore, the object embedding features of at least one visual object may be input into an object routing model, and each object embedding feature may be analyzed and processed according to the object routing model to predict the distribution probability of each visual object in each object region, thereby arranging each visual object in the object region with the highest probability, and routing each visual object to the corresponding object region, and the object routing model may be any model capable of analyzing the distribution probability of the object embedding features of each visual object in each object region. Wherein, each object region is an object region included in the target processing result to be obtained.

[0070] For example, the target visual task includes three visual objects, a text prompt message "Three little girls posing together on a swing, elegantly dressed" and a reference image of three children swinging on a swing. The face positions of the three children in the reference image are the object regions to be predicted, and the distribution probability of the three visual objects in each object region is predicted.

[0071] In an optional implementation of this embodiment, the object embedding model may include an object recognition module, an object encoding module, and a fusion module, and extracts object embedding features of at least one visual object, including:

[0072] Obtaining object identity features of at least one visual object using an object recognition module of the embedding model;

[0073] Obtaining object structural features of at least one visual object using an object encoding module of the embedding model;

[0074] An object embedding feature of at least one visual object is obtained according to an object identity feature of at least one visual object and an object structure feature of at least one visual object by using a fusion module of the embedding model.

[0075] Among them, the object recognition module is a network model for extracting the object identity features of visual objects, the object encoding module is a network model for extracting the object structure features of visual objects, and the fusion module is a network model that fuses the object identity features and the object structure features to obtain the object embedding features.

[0076] In actual implementation, at least one visual object can be input into an object recognition module, which extracts features from each visual object and outputs object identity features corresponding to each visual object. At the same time, at least one visual object can be input into an object encoding module, which encodes each visual object and outputs object structure features corresponding to each visual object. Then, the object identity features and object structure features of each visual object are input into a fusion module, and for each visual object, the fusion module fuses the corresponding object identity features and object structure features to obtain the object embedding features corresponding to the visual object.

[0077] It should be noted that the object recognition module included in the object embedding model can identify and obtain the object identity features, and combine the object structure features obtained by the object encoding module to determine the object embedding features of the visual object. In addition to considering the object identity features, the object structure features are introduced to better preserve the shape and texture details, enhance the fidelity of the visual object in the embedding feature extraction process, and enable the visual object to have higher object fidelity and editability in the subsequent visual task processing process.

[0078] Of course, in actual implementation, at least one visual object may be directly input into the embedding model to extract the object embedding features of each visual object, and this specification does not limit this.

[0079] In an optional implementation of this embodiment, the object recognition module includes a structure recognition network and an identity recognition network; using the object recognition module embedded in the model to obtain the object identity feature of at least one visual object includes:

[0080] Inputting at least one visual object into a structure recognition network to obtain a spatial structure representation of the at least one visual object;

[0081] The spatial structure representation of at least one visual object is input into an identity recognition network to obtain an object identity feature of the at least one visual object.

[0082] Among them, the structure recognition network is a feature extraction model that can identify the backbone points of visual objects. The spatial structure representation obtained by the structure recognition network can effectively represent the uniqueness of each visual object, so that different visual objects can be accurately distinguished. For example, when the visual object is a face, the structure recognition network can be a face recognition model.

[0083] In addition, the identity recognition network is a feedforward neural network that can further perform feature conversion on the spatial structure representation, converting the spatial structure representation into the required object identity features. For example, the identity recognition network can be a multi-layer perceptron (MLP).

[0084] In actual implementation, the structure recognition network includes a feature extraction layer and a fully connected layer. The feature extraction layer is a model layer before the fully connected layer, and the feature extraction layer can extract discriminative features with spatial structure. Specifically, at least one visual object is input into the structure recognition network, and the structure recognition network recognizes and extracts each visual object at a set size, and obtains the discriminative features output by the feature extraction layer (the penultimate layer, that is, the model layer before the fully connected layer) of the structure recognition network as the spatial structure representation of each visual object to retain the spatial information of the degree of enhancement related to the identity of the object. The spatial structure representation of each visual object is input into the MLP network to obtain the object identity features of each visual object. Among them, the set size is the pre-configured size of the collected object identity features. In order to obtain deeper features that are more relevant to the identity, feature sampling can be performed at a deeper scale, such as the set size can be 1 / 16 of the original image.

[0085] In the embodiments of this specification, a multi-layer perceptron can be applied to the feature extraction layer of the structure recognition network to obtain discriminative features with spatial structure as object identity features, thereby enhancing the fidelity of visual objects in the embedded feature extraction process.

[0086] In an optional implementation of this embodiment, the object encoding module includes an image encoder and a feature combination network; before obtaining the object structure feature of at least one visual object using the object encoding module of the embedded model, the method further includes:

[0087] Sampling at least one visual object at different sizes to obtain structural representations of at least one visual object at different sizes;

[0088] Accordingly, the object encoding module of the embedding model is used to obtain the object structure feature of at least one visual object, including:

[0089] Inputting at least one visual object into an image encoder to obtain an object encoding feature of the at least one visual object;

[0090] The object encoding features of at least one visual object and the structural representations of at least one visual object at different sizes are input into a feature combination network to obtain the object structural features of the at least one visual object.

[0091] Among them, the image encoder is a model that can encode the image features of the input visual object to achieve cross-modal conversion from image to language. The object encoding features obtained based on the image encoder can represent the local features of the visual object in the language mode. For example, the image encoder can be CLIP. The feature combination network can be a feature conversion model that can fuse the object encoding features and structural representations at different scales. In an optional implementation, the feature combination network can also be MLP.

[0092] In actual implementation, at least one visual object can be sampled at different sizes to obtain structural representations of at least one visual object at different sizes. The different sizes should be larger than the sampling size when the structure recognition network obtains the object identity features, so as to obtain shallow features of the visual object, which can contain more texture details and are more relevant to the object identity. For example, the different sizes can be 1 / 2, 1 / 4, 1 / 8, etc.

[0093] Specifically, other feature extraction models or sampling algorithms can be used to sample at least one visual object at different sizes to obtain structural representations of at least one visual object at different sizes, or at least one visual object can be sampled at different sizes based on the above-mentioned structure recognition network to obtain structural representations of at least one visual object at different sizes.

[0094] For example, at least one visual object is input into the structure recognition network, and the structure recognition network samples each visual object at 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original image. The features obtained by sampling at the 1 / 16 scale are used as spatial structure representations and input into the MLP network connected thereafter to obtain object identity features. The structural representations obtained by sampling at the 1 / 2, 1 / 4, and 1 / 8 scales are used as shallow features and input into the MLP network connected thereafter together with the image encoding features (i.e., local features) obtained by CLIP to obtain object structure features.

[0095] In the embodiments of the present specification, object coding features obtained by encoding with an image encoder and structural representations of at least one visual object at different sizes are combined to obtain object structural features, thereby retaining more texture details and improving object fidelity, so that the visual object can have higher object fidelity and editability in subsequent visual task processing.

[0096] In an optional implementation of this embodiment, the fusion module includes a first attention mechanism layer, a second attention mechanism layer, and a feature conversion layer; using the fusion module of the embedding model, obtaining an object embedding feature of at least one visual object according to an object identity feature of at least one visual object and an object structure feature of at least one visual object, including:

[0097] Inputting object identity features of at least one visual object into a first attention mechanism layer to obtain an identity feature representation sequence;

[0098] Inputting the identity feature representation sequence and the object structure feature of at least one visual object into the second attention mechanism layer to obtain the structure feature representation sequence;

[0099] The identity feature representation sequence and the structure feature representation sequence are input into the feature conversion layer to obtain the object embedding feature of at least one visual object.

[0100] In actual implementation, the attention mechanism can be used to realize the fusion of object identity features and object structure features. Specifically, the object identity features of at least one visual object can be input into the first attention mechanism layer. The first attention layer performs cross-attention calculation on the information contained in the object identity features of any visual object to obtain an identity feature representation sequence; then, the identity feature representation sequence and the object structure features of at least one visual object are superimposed and input into the second attention mechanism layer, so that the second attention mechanism layer performs cross-attention calculation on the identity feature representation sequence and the object structure features of at least one visual object to obtain a structure feature representation sequence. Afterwards, the identity feature representation sequence and the structure feature representation sequence are input into the feature conversion layer to obtain the object embedding features of at least one visual object. Among them, the feature conversion layer is, such as FFN.

[0101] It should be noted that two attention mechanism layers are used to process object identity features and object structure features respectively, so as to separate object identity features and object structure features, and prevent overfitting of object structure features and loss of object identity features, thereby ensuring higher editability. In addition, when processing object structure features, the identity feature representation sequence of object identity features is combined, so that the extracted object embedding features can be more dependent on object identity features, and better balance object similarity and object editability.

[0102] In an optional implementation of this embodiment, determining spatial distribution information of at least one visual object according to an object embedding feature of at least one visual object includes:

[0103] Performing attention calculation on the object embedding features of at least one visual object respectively to obtain an object representation corresponding to the at least one visual object;

[0104] Inputting the object embedding feature of at least one visual object into the object routing model to obtain an object position mask of at least one visual object, wherein one visual object corresponds to one object position mask;

[0105] Spatial distribution information of at least one visual object is obtained based on the object representation and the object position mask of the at least one visual object.

[0106] Among them, the object routing model is a model that can adaptively assign each visual object to a corresponding object area, and ensure that each visual object is routed and only routed to a corresponding object area, and each visual object is embedded into the context space using the object routing model.

[0107] It should be noted that after the object embedding features are extracted, the corresponding object embedding features are extracted from each visual object, and then the object embedding features of each visual object can be analyzed to determine in which object area each visual object should be distributed.

[0108] In actual implementation, the object embedding features of at least one visual object can be input into the object routing model. The object routing model can calculate an n-dimensional probability distribution for each object area based on the object embedding features of each visual object, where n is the number of visual objects. That is, for each object area, the probability of each visual object in the object area is predicted, and the object area with the largest probability is selected to embed the object area to obtain the object position mask of the object area.

[0109] In addition, in order to avoid associating multiple visual objects with one object area, a cross-attention mechanism can be introduced. Specifically, the object embedding features of at least one visual object can be respectively calculated to obtain the object representation corresponding to at least one visual object. For any visual object, the obtained object representation and the object position mask can be multiplied to obtain the spatial distribution information of the visual object, that is, the object area where the visual object is located is 1, and other areas are 0. Since the spatial distribution information of a visual object is 1 in the object area where the visual object is located and 0 in other areas, the spatial distribution information of each visual object can be fused and superimposed to obtain the spatial distribution information of at least one visual object as the input of the subsequent target visual model for visual task processing.

[0110] In the embodiments of this specification, an object routing model is used to route each visual object to a corresponding object area, and an attention mechanism is used to constrain an object area to focus on only one visual object, thereby avoiding the mixing problem of different visual objects, achieving the allocation of multiple visual objects, and improving the accuracy and flexibility of visual task processing.

[0111] Step 206: According to the spatial distribution information of at least one visual object, at least one visual object is assigned to a corresponding object area using a target visual processing model to obtain a target processing result of the target visual task.

[0112] It should be noted that the target visual processing model is a pre-trained visual processing model, which is used to process the target visual task and output the target processing result of the target visual task. The target visual processing model is a large model, such as a diffusion model.

[0113] In actual implementation, the target visual task may include, in addition to at least one visual object, a reference image and text prompt information. The text prompt information is used to instruct the target visual processing model to synthesize at least one visual object into the reference image in accordance with the corresponding processing method based on the spatial distribution information of at least one visual object to obtain the target processing result.

[0114] In an optional implementation of this embodiment, the task data further includes a reference image and / or text prompt information; and according to the spatial distribution information of at least one visual object, at least one visual object is assigned to a corresponding object area using a target visual processing model to obtain a target processing result of the target visual task, including:

[0115] According to the reference image and / or text prompt information, and the spatial distribution information of at least one visual object, the target visual processing model is used to assign at least one visual object to the object area corresponding to the reference image to obtain the target processing result of the target visual task.

[0116] The text prompt information is used to describe the visual object, and there is no restriction on the format and content of the text prompt information in the embodiments of this specification.

[0117] It should be noted that the reference image, text prompt information and the spatial distribution information of at least one visual object can be input into the trained target visual processing model (diffusion model). The target visual processing model can assign each visual object to the object area corresponding to the reference image according to the spatial distribution information to obtain the target processing result of the target visual task.

[0118] By applying the solution of the embodiments of this specification, the spatial distribution information of at least one visual object is input into the target visual processing model as a prompt information, so as to realize the customized generation of multiple visual objects. There is no need to restrict the text prompts and object layout of the target visual processing model, and it does not affect the integrity of the identity of each visual object, the diversity of the generated target processing results and the flexibility of the prompt design. It can be compatible with most large models and achieves good scalability and universal compatibility.

[0119] The following combination Figure 3 , taking the application of the visual task processing method provided in this specification in the image generation scenario as an example, the visual task processing method is further explained. Figure 3 A schematic diagram of a processing process of a visual task processing method provided by an embodiment of this specification is shown. Figure 4 A schematic diagram of a visual task processing interface provided by an embodiment of this specification is shown. Figure 5 A schematic diagram of another visual task processing interface provided by an embodiment of the present specification is shown.

[0120] like Figure 4 As shown, the user can click the "Upload Face" control in the visual task processing interface, and the control may display the upload prompt information "Please upload the face to be synthesized here" by default. The user can select at least one face of the image to be synthesized as the visual object and click to upload; and the user can click the "Task Information" control to enter text prompt information, and the control may display the input prompt "Please enter the requirements for the synthesized image here" by default; in addition, the visual task processing interface also displays a synthetic image template, such as Template 1-Template N, and the user can select any template as a reference image. Assume that the user uploaded Face 1, Face 2, and Face 3 as visual objects, entered "Three little girls posing together on a swing, dressed elegantly" as the text prompt information, and selected Template 1 as the reference image in the visual processing interface.

[0121] like Figure 3As shown, the visual task processing platform can obtain face 1, face 2, and face 3, input face 1, face 2, and face 3 into the face recognition module, sample face 1, face 2, and face 3 at a size of 1 / 16, respectively, and obtain spatial structure representation 1 of face 1, spatial structure representation 2 of face 2, and spatial structure representation 3 of face 3, and input the spatial structure representation 1, spatial structure representation 2, and spatial structure representation 3 into the identity recognition network (MLP1) to obtain face identity feature 1 of face 1, face identity feature 2 of face 2, and face identity feature 3 of face 3.

[0122] Moreover, after face 1, face 2, and face 3 are input into the face recognition module, face 1, face 2, and face 3 can be sampled at 1 / 2, 1 / 4, and 1 / 8 sizes respectively to obtain shallow facial feature 1 of face 1, shallow facial feature 2 of face 2, and shallow facial feature 3 of face 3.

[0123] Face 1, face 2, and face 3 are input into the image encoder (CLIP) to obtain face coding feature 1 of face 1, face coding feature 2 of face 2, and face coding feature 3 of face 3; face shallow feature 1, face shallow feature 2, face shallow feature 3, as well as face coding feature 1, face coding feature 2, and face coding feature 3 are input into the feature combination network (MLP2); face shallow feature 1 and face coding feature 1 are fused to obtain face structure feature 1 of face 1; face shallow feature 2 and face coding feature 2 are fused to obtain face structure feature 2 of face 2; face shallow feature 3 and face coding feature 3 are fused to obtain face structure feature 3 of face 3.

[0124] Input face identity feature 1, face identity feature 2, and face identity feature 3 into the first attention mechanism layer to obtain identity feature representation sequence 1 for face 1, identity feature representation sequence 2 for face 2, and identity feature representation sequence 3 for face 3.

[0125] The identity feature representation sequence 1, identity feature representation sequence 2, identity feature representation sequence 3, as well as face structure feature 1, face structure feature 2, face structure feature 3 are input into the second attention mechanism layer, and the structural feature representation sequence 1 of face 1 is obtained based on the identity feature representation sequence 1 and face structure feature 1, the structural feature representation sequence 2 of face 2 is obtained based on the identity feature representation sequence 2 and face structure feature 2, and the structural feature representation sequence 3 of face 3 is obtained based on the identity feature representation sequence 3 and face structure feature 3.

[0126] The identity feature representation sequence 1, identity feature representation sequence 2, identity feature representation sequence 3, as well as the structure feature representation sequence 1, structure feature representation sequence 2, structure feature representation sequence 3 are input into the feature conversion layer (FFN) to obtain the face embedding feature 1 of face 1, the face embedding feature 2 of face 2, and the face embedding feature 3 of face 3.

[0127] Input face embedding feature 1 into attention module 1 to obtain face representation 1 of face 1, input face embedding feature 2 into attention module 2 to obtain face representation 2 of face 2, input face embedding feature 3 into attention module 3 to obtain face representation 3 of face 3.

[0128] Input face embedding feature 1, face embedding feature 2, and face embedding feature 3 into the face routing model, predict the face with the highest probability corresponding to the three object regions included in the reference image, assume that the object region 1 of the reference image has the highest probability of face 1, embed face 1 into object region 1, and obtain the region mask of face 1 Figure 1 ; The most likely face in object area 2 is face 2, so face 2 is embedded in object area 2 to obtain the area mask of face 2 Figure 2 ; The most likely object in object region 3 is face 3, so face 3 is embedded in object region 3 to obtain the region mask of face 3 Figure 3 .

[0129] The face representation 1 and the region mask Figure 1 Multiply, face representation 2 and region mask Figure 2 Multiply, face representation 3 and region mask Figure 3 Multiply them, and superimpose the multiplied results to obtain spatial distribution information.

[0130] The spatial distribution information, the reference image and the text prompt information are input into the diffusion model. The diffusion model assigns face 1 to object area 1 of the reference image, assigns face 2 to object area 2 of the reference image, and assigns face 3 to object area 3 of the reference image, so as to superimpose face 1, face 2 and face 3 onto the reference image, generate a target image as the target processing result of the target visual task, and display the target image to the front end, such as Figure 5 shown.

[0131] By applying the scheme of the embodiments of this specification, face embedding features of multiple faces are extracted, and the spatial distribution information of multiple faces is determined, and the multiple faces are allocated to corresponding object areas using a diffusion model to generate a target image. The multiple faces are adaptively combined and distributed to corresponding object areas, and multiple faces are allocated and managed to achieve customized generation of multiple faces without restricting the text prompts and layout of the diffusion model, without requiring manual intervention, without affecting the integrity of the faces, the diversity of the target image and the flexibility of the text prompts, thereby improving the accuracy of the generated images, being compatible with most diffusion models, and achieving good scalability and universal compatibility.

[0132] See also Figure 6 , Figure 6 A flowchart of a method for training a visual processing model provided according to an embodiment of the present specification is shown, which specifically includes the following steps 602-606.

[0133] Step 602: Acquire training data for a training task, wherein the training data includes at least one sample object and a corresponding result label.

[0134] The training data is the historical data of the processing results corresponding to the training task, which is used to train the initial visual model. The training data includes at least one sample object and the corresponding result label. The result label is the processing result that is expected to be obtained, which is used to determine the prediction loss of the initial visual model for the training task, so as to adjust the model training.

[0135] As an example, multiple sample images can be obtained, and sample objects can be extracted from the sample images. The sample images are the result labels corresponding to the extracted sample objects. In this case, the sample objects used for training come from the result labels. As another example, sample objects and historical synthetic images corresponding to the sample objects can also be obtained, and the historical synthetic images are used as the result labels. In this case, the sample objects used for training and the result labels are different images.

[0136] Step 604: extracting a sample object embedding feature of at least one sample object, and determining sample space distribution information of at least one sample object according to the sample object embedding feature of the at least one sample object.

[0137] It should be noted that the specific implementation process of "extracting sample object embedding features of at least one sample object, and determining the sample spatial distribution information of at least one sample object based on the sample object embedding features of at least one sample object" in the model training stage is similar to that of "extracting object embedding features of at least one visual object, and determining the spatial distribution information of at least one visual object based on the object embedding features of at least one visual object" in the above-mentioned task processing stage (that is, the application reasoning stage), which will not be repeated here.

[0138] In an optional implementation of this embodiment, the training data includes first sample data and second sample data, and the second sample data includes at least two sample objects; before extracting the sample object embedding feature of at least one sample object, the method further includes:

[0139] Using the first sample data, single-object training is performed on the initial embedding model and the initial visual processing model to obtain an updated visual processing model and a trained object embedding model;

[0140] Accordingly, extracting a sample object embedding feature of at least one sample object, and determining sample space distribution information of at least one sample object according to the sample object embedding feature of at least one sample object, includes:

[0141] Using the trained object embedding model, obtaining sample object embedding features of at least two sample objects;

[0142] The sample object embedding features of at least two sample objects are input into the initial routing model to obtain sample space distribution information of at least two sample objects.

[0143] In actual implementation, the training data includes first sample data and second sample data. The first sample data refers to the original data used for single-object training of the initial embedding model and the initial visual processing model, and the first sample data includes a single sample object. The second sample data refers to the data used for multi-object routing training of the initial routing model, and includes at least two sample objects, that is, the first sample data is single-object sample data, and the second sample data is multi-object sample data.

[0144] In actual implementation, the first sample data can be used to perform the first stage training on the initial embedding model and the initial visual processing model, that is, the single-object training is first performed on the initial embedding model and the initial visual processing model based on the single sample object. After the training is completed, the second stage training of multi-object routing is performed on the initial routing model based on the multi-object sample data, that is, the multi-object adjustment stage.

[0145] In the embodiments of the present specification, the overall model training is divided into two stages, a single object training stage and a multi-object adjustment stage. The single object training stage mainly trains the initial embedding model so that the trained object embedding model can more accurately extract the object embedding features of the visual object; the multi-object adjustment stage mainly trains the initial routing module so that the trained object routing model can accurately route each visual object to the corresponding object area. After completing the two-stage training, single-object customized generation can be achieved, and multi-object customized generation can also be achieved without restricting the text prompts and layout of the target visual processing model and without manual intervention, thereby improving the accuracy of the training model.

[0146] In an optional implementation of this embodiment, the initial embedding model includes an initial recognition module, an initial encoding module, a first discarding module and an initial fusion module, and the first sample data includes a single sample object and a corresponding single object result label; using the first sample data, the initial embedding model and the initial visual processing model are trained on a single object to obtain an updated visual processing model and a trained object embedding model, including:

[0147] Inputting a single sample object into an initial recognition module to obtain a sample object identity feature of the single sample object;

[0148] Using the initial encoding module, a sample object structure feature of a single sample object is obtained, and the sample object structure feature is input into a first discarding module to obtain an updated object structure feature, wherein the first discarding module is used to randomly discard part of the features in the sample object structure;

[0149] Input the sample object identity feature and the updated object structure feature into the initial fusion module to obtain the sample object embedding feature of the single sample object;

[0150] The sample object embedding feature of the single sample object is input into the initial visual processing model to obtain a second prediction result, and the model parameters of the initial embedding model and the initial visual processing model are adjusted according to the single object result label and the second prediction result until the second training stop condition is reached to complete the single object training.

[0151] The first sample data includes a single sample object and a corresponding single object result label, and the single object result label refers to the processing result that the first sample data is expected to obtain. As an example, multiple first sample images can be obtained, and the first sample image is a historical image including a face. A face is extracted from the first sample image, and the face is used as the first sample data, and the first sample image is the corresponding single result label.

[0152] It should be noted that since structural features may be coupled with other redundant information, the model may overfit these other redundant information, where the other redundant information may include information unrelated to identity, such as annotations, posture, etc.; features unrelated to the object, such as lighting, shadows, etc. Therefore, in actual implementation, during the model training stage, a first discard module can be introduced into the initial embedding model, and the first discard module can be used to randomly discard some structural features to reduce the impact of structural features.

[0153] In actual implementation, after using the initial encoding module to obtain the sample object structure features of a single sample object, the sample object structure features can be input into the first discarding module. The first discarding module randomly discards some features in the sample object structure based on the discarding probability to obtain updated object structure features, which are then input into the initial fusion module for feature fusion to obtain sample object embedding features.

[0154] Since the initial routing model is not adjusted in the single-object training phase, the sample object embedding features of the single-sample object can be directly input into the initial visual processing model to obtain the second prediction result corresponding to the single-sample object, and then the prediction loss is calculated based on the single-object result label and the second prediction result, and the model parameters of the initial embedding model and the initial visual processing model are reversely adjusted until the second training stop condition is reached, the single-object training is completed, and the trained object embedding module is obtained. After the training is completed, when the trained object embedding module is applied, the first discard module is not used, that is, the structural features are not discarded.

[0155] In the embodiments of this specification, when performing single-object training on the initial embedding model, a first discarding module is introduced to randomly discard some structural features to avoid overfitting in the model training process and improve the accuracy of model training, thereby improving the accuracy of extracting object embedding features and ensuring the accuracy of task processing.

[0156] In an optional implementation of this embodiment, the initial fusion module includes a first initial attention mechanism layer, a second initial attention mechanism layer, a second discarding module and an initial feature conversion layer; the sample object identity feature and the updated object structure feature are input into the initial fusion module to obtain the sample object embedding feature of the single sample object, including:

[0157] Inputting the sample object identity feature into the first initial attention layer to obtain a sample identity feature representation sequence;

[0158] Input the sample identity feature representation sequence and the updated object structure feature into the second initial attention mechanism layer to obtain the sample structure feature representation sequence;

[0159] The sample structure feature representation sequence is input into the second discarding module, and the output result of the second discarding module is fused with the sample identity feature representation sequence to obtain the sample object embedding feature of the single sample object, wherein the second discarding module is used to discard the updated object structure feature based on the set probability.

[0160] It should be noted that a second discarding module can also be introduced into the initial fusion module to discard the updated object structure features based on a set probability. In actual implementation, a second discarding module can be added after the second initial attention mechanism layer, and the second discarding module discards the updated object structure features based on a set probability, wherein the set probability can be configured based on demand, such as 0.5, that is, the updated object structure features are discarded once and not discarded once.

[0161] If the updated object structure features are discarded, it means that all object structure features are discarded. At this time, the sample object embedding features of the single sample object are obtained only based on the sample object identity features. If the updated object structure features are not discarded, the sample object identity features and the sample object structure features are combined to obtain the sample object embedding features of the single sample object, and the sample object identity features and the sample object structure features are separated.

[0162] In the embodiments of the present specification, during the model training stage, a second discarding module is further introduced to discard structural features based on a set probability, separate the sample object identity features and the sample object structural features, decouple the identity features and the structural features, and make the trained object embedding model more dependent on the identity features, thereby better balancing object similarity and object editability.

[0163] In an optional implementation of this embodiment, the sample object embedding features of at least two sample objects are input into the initial routing model to obtain sample space distribution information of at least two sample objects, including:

[0164] Performing attention calculations on sample object embedding features of at least two sample objects respectively to obtain object representations corresponding to the at least two sample objects;

[0165] Inputting sample object embedding features of at least two sample objects into an initial routing model to obtain sample position masks of at least two sample objects respectively, wherein one sample object corresponds to one sample position mask;

[0166] Sample spatial distribution information of at least two sample objects is obtained according to the object representations and sample position masks of at least two sample objects.

[0167] It should be noted that the specific process of obtaining the sample spatial distribution information in the above-mentioned training phase is the same as the specific process of obtaining the spatial distribution information in the above-mentioned task processing phase (ie, the application reasoning phase), and will not be repeated here.

[0168] Step 606: Train the initial visual processing model based on the sample space distribution information of at least one sample object in combination with the result label to obtain a trained target visual processing model.

[0169] In an optional implementation of this embodiment, the second sample data further includes a multi-object result label; training the initial visual processing model according to the sample space distribution information of at least one sample object in combination with the result label to obtain a trained target visual processing model, including:

[0170] Obtaining a first prediction processing result by updating the visual processing model according to sample space distribution information of at least two sample objects;

[0171] According to the first prediction processing result and the multi-object result labels, the model loss is determined, and the model parameters of the initial routing model and the updated visual processing model are adjusted according to the model loss until the first training stop condition is reached, thereby obtaining the trained object routing model and the target visual processing model.

[0172] The second sample data includes at least two sample objects and corresponding multi-object result labels, and the multi-object result labels refer to the processing results that the at least two sample objects hope to obtain. As an example, multiple second sample images can be obtained, and the second sample images are historical images including multiple faces. Multiple faces are extracted from the second sample images, and the multiple faces are used as the second sample data, and the first sample image is the corresponding multi-result label.

[0173] It should be noted that the trained object embedding model can be used to obtain sample space distribution information of at least two sample objects, and then based on the sample space distribution information of at least two sample objects, the updated visual processing model is used to obtain a first prediction processing result for at least two sample objects. Based on the first prediction processing result and the multi-object result label, the model loss is determined, and the initial routing model and the updated visual processing model are adjusted until the first training stop condition is reached, and the trained object routing model and the target visual processing model are obtained, completing the adjustment of the multi-object stage.

[0174] The first training stop condition and the second training stop condition may be that the loss value is less than the loss value threshold, or the number of iterations reaches the number threshold, or the loss value threshold and the number threshold may be combined to determine whether the training stop condition is reached. The first training stop condition and the second training stop condition may be the same or different.

[0175] The loss value threshold is the critical value of the loss value. When the loss value is greater than or equal to the loss value threshold, it means that there is still a certain deviation between the predicted result and the actual result, and the model parameters still need to be adjusted. At this time, it is determined that the first training stop condition or the second training stop condition has not been met; when the loss value is less than the loss value threshold, it means that the predicted result is close enough to the actual result, and the training can be stopped. At this time, it can be determined that the first training stop condition or the second training stop condition has been met. The value of the loss value threshold is selected according to the actual situation.

[0176] In addition, the number threshold is set according to the actual situation. When the number of iterations reaches the number threshold, it means that the number of training times is sufficient. At this time, the prediction result is very close to the real result, and the training can be stopped.

[0177] In the embodiments of this specification, the specific training situation of the model can be judged according to the loss value, and when the training fails, the model parameters can be reversely adjusted according to the loss value to improve the processing capability of the model, with a high training rate and good training effect.

[0178] In an optional implementation of this embodiment, determining the model loss according to the first prediction processing result and the multi-object result labels includes:

[0179] Detecting the object boundaries of each sample object in the multi-object result labels;

[0180] Generate a label position mask for each sample object according to the object boundary of each sample object;

[0181] A routing loss is calculated according to the sample position mask and the label position mask, a prediction loss is calculated according to the first prediction processing result and the multi-object result labels, and a model loss is determined according to the routing loss and the prediction loss.

[0182] In actual implementation, models, algorithms or programming techniques can be used to detect the object boundaries of each sample object in the multi-object result label, generate a label position mask for each sample object, and then calculate the routing loss based on the sample position mask and the label position mask, calculate the prediction loss based on the first prediction processing result and the multi-object result label, superimpose the routing loss and the prediction model as the model loss, adjust the initial routing model and update the model parameters of the visual processing model.

[0183] It should be noted that the routing loss is introduced to supervise the learning of the initial routing model, and the loss of each object area in the multi-object result label is learned to ensure that each sample object is routed and only routed to one object area, so that the trained object routing module can avoid the mixing problem of different visual objects, realize the allocation of multiple visual objects, and improve the accuracy and flexibility of visual task processing.

[0184] By applying the scheme of the embodiments of this specification, the initial visual processing model can be trained based on the sample space distribution information of at least one sample object in combination with the result label to obtain a trained target visual processing model. The sample space distribution information can be used as an information reference for the initial visual processing model so that the initial visual processing model can learn to route at least one sample object and assign at least one sample visual object to a corresponding object area. This allows the trained target visual model to achieve customized generation of single and multiple visual objects without restricting the text prompts and layout of the target visual processing model.

[0185] The following combination Figure 7 , taking the application of the training method of the visual processing model provided in this specification in the image generation scene as an example, the training method of the visual processing model is further explained. Among them, Figure 7 A schematic diagram of the processing process of a visual processing model training method provided by an embodiment of the present specification is shown.

[0186] Obtain sample face 1, input sample face 1 into the initial face recognition module, sample face 1 is sampled at a size of 1 / 16, sample space structure representation 1 of sample face 1 is obtained, and sample space structure representation 1 is input into the initial identity recognition network (MLP1) to obtain sample face identity feature 1 of sample face 1. Furthermore, after inputting sample face 1 into the initial face recognition module, sample face 1 can also be sampled at sizes of 1 / 2, 1 / 4, and 1 / 8, respectively, to obtain sample face shallow feature 1 of sample face 1.

[0187] The sample face 1 is input into the initial image encoder (CLIP) to obtain the sample face coding feature 1 of the sample face 1, the sample face shallow feature 1 and the sample face coding feature 1 are input into the initial feature combination network (MLP2), the sample face shallow feature 1 and the sample face coding feature 1 are fused to obtain the sample face structural feature 1 of the sample face 1.

[0188] The sample face identity feature 1 is input into the initial first attention mechanism layer to obtain the sample identity feature representation sequence 1 of the sample face 1. The sample face structure feature 1 is input into the first discard module to obtain the updated face structure feature 1. The sample identity feature representation sequence 1 and the updated face structure feature 1 are input into the second attention mechanism layer, and the sample structure feature representation sequence 1 of the sample face 1 is obtained based on the sample identity feature representation sequence 1 and the updated face structure feature 1.

[0189] The sample structure feature representation sequence 1 is input into the second discarding module to obtain a discarding result. The sample identity feature representation sequence 1 and the discarding result are input into the initial feature conversion layer (FFN) to obtain the sample face embedding feature 1 of the sample face 1.

[0190] The sample face embedding feature 1 is input into the initial diffusion model to obtain the predicted image. According to the label image and the predicted image result, the model parameters are adjusted to complete the single object training. After the single object training is completed, the face embedding model is obtained.

[0191] like Figure 7 As shown, sample face 2, sample face 3, and sample face 4 can be obtained, and sample face 2, sample face 3, and sample face 4 can be input into the face recognition module. Sample face 2, sample face 3, and sample face 4 are sampled at a size of 1 / 16 respectively to obtain sample spatial structure representation 2 of sample face 2, sample spatial structure representation 3 of sample face 3, and sample spatial structure representation 4 of sample face 4. The sample spatial structure representation 2, sample spatial structure representation 3, and sample spatial structure representation 4 are input into the identity recognition network (MLP1) to obtain sample face identity feature 2 of sample face 2, sample face identity feature 3 of sample face 3, and sample face identity feature 4 of sample face 4.

[0192] Moreover, after inputting sample face 2, sample face 3, and sample face 4 into the face recognition module, sample face 2, sample face 3, and sample face 4 can be sampled at sizes of 1 / 2, 1 / 4, and 1 / 8, respectively, to obtain sample face shallow feature 2 of sample face 2, sample face shallow feature 3 of sample face 3, and sample face shallow feature 4 of sample face 4.

[0193] Input sample face 2, sample face 3, and sample face 4 into the image encoder (CLIP) to obtain sample face coding feature 2 of sample face 2, sample face coding feature 3 of sample face 3, and sample face coding feature 4 of sample face 4; input sample face shallow feature 2, sample face shallow feature 3, and sample face shallow feature 4 as well as sample face coding feature 2, sample face coding feature 3, and sample face coding feature 4 into the feature combination network (MLP2); fuse the sample face shallow feature 2 and the sample face coding feature 2 to obtain sample face structural feature 2 of sample face 2; fuse the sample face shallow feature 3 and the sample face coding feature 3 to obtain the sample face structural feature 3 of sample face 3; fuse the sample face shallow feature 4 and the sample face coding feature 4 to obtain the sample face structural feature 4 of sample face 4.

[0194] Input sample face identity feature 2, sample face identity feature 3, and sample face identity feature 4 into the first attention mechanism layer to obtain sample identity feature representation sequence 2 of sample face 2, sample identity feature representation sequence 3 of sample face 3, and sample identity feature representation sequence 4 of sample face 4.

[0195] The sample facial structure feature 2, the sample facial structure feature 3, and the sample facial structure feature 4 are input into the first discarding module to obtain the updated facial structure feature 2, the updated facial structure feature 3, and the updated facial structure feature 4.

[0196] The sample identity feature representation sequence 2, the sample identity feature representation sequence 3, the sample identity feature representation sequence 4, as well as the updated face structure feature 2, the updated face structure feature 3, and the updated face structure feature 4 are input into the second attention mechanism layer, and the sample structure feature representation sequence 2 of the sample face 2 is obtained based on the sample identity feature representation sequence 2 and the updated face structure feature 2, the sample structure feature representation sequence 3 of the sample face 3 is obtained based on the sample identity feature representation sequence 3 and the updated face structure feature 3, and the sample structure feature representation sequence 4 of the sample face 4 is obtained based on the sample identity feature representation sequence 4 and the updated face structure feature 4.

[0197] The sample structure feature representation sequence 2, the sample structure feature representation sequence 3, and the sample structure feature representation sequence 4 are input into the second discarding module to obtain a discarding result.

[0198] The sample identity feature representation sequence 2, the sample identity feature representation sequence 3, the sample identity feature representation sequence 4 and the discarded result are input into the feature conversion layer (FFN) to obtain the sample face embedding feature 2 of the sample face 2, the sample face embedding feature 3 of the sample face 3, and the sample face embedding feature 4 of the sample face 4.

[0199] The sample face embedding feature 2 is input into the initial attention module 1 to obtain the sample face representation 2 of the sample face 2; the sample face embedding feature 3 is input into the initial attention module 2 to obtain the sample face representation 3 of the sample face 3; the sample face embedding feature 4 is input into the initial attention module 3 to obtain the sample face representation 4 of the sample face 4.

[0200] Input sample face embedding feature 2, sample face embedding feature 3, and sample face embedding feature 4 into the initial face routing model, predict the face with the highest probability corresponding to the three sample object regions included in the sample image, assume that the sample object region 2 of the sample image has the highest probability of being sample face 2, embed sample face 2 into sample object region 2, and obtain the sample region mask of sample face 2 Figure 2 The sample object region 3 with the highest probability is sample face 3, and sample face 3 is embedded in sample object region 3 to obtain a sample region mask for sample face 3 Figure 3 The sample object region 4 with the highest probability is the sample face 4, and the sample face 4 is embedded in the sample object region 4 to obtain the sample region mask of the sample face 4 Figure 4 .

[0201] The sample face representation 2 and the sample region mask Figure 2 Multiply the sample face representation 3 and the sample region mask Figure 3 Multiply the sample face representation 4 and the sample region mask Figure 4 Multiply them, and superimpose the multiplied results to obtain the sample space distribution information.

[0202] The sample spatial distribution information, sample image and sample text prompt information are input into the updated diffusion model after the above-mentioned single-object training. The updated diffusion model assigns sample face 2 to sample object area 2 of the sample image, assigns sample face 3 to sample object area 3 of the sample image, and assigns sample face 4 to sample object area 4 of the sample image. Sample face 2, sample face 3 and sample face 4 are superimposed on the sample image to generate a predicted image as the prediction result of the training task.

[0203] Detect the boundary of sample face 2 in the label image and generate label position mask 2; detect the boundary of sample face 3 in the label image and generate label position mask 3; detect the boundary of sample face 4 in the label image and generate label position mask 4.

[0204] Based on the sample area mask Figure 2 , sample area mask Figure 3 , sample area mask Figure 4 , and label position mask 2, label position mask 3, and label position mask 4 calculate the routing loss, and calculate the prediction loss according to the predicted image and the label image, determine the model loss according to the routing loss and the prediction loss, adjust the model parameters according to the model loss, and obtain the trained face routing model and diffusion model.

[0205] By applying the scheme of the embodiments of this specification, the initial diffusion model can be trained according to the distribution information of the faces in combination with the result labels to obtain a trained diffusion model. The distribution information of the faces is used as a reference for the diffusion model, so that the diffusion model can learn to route multiple faces and realize customized generation of multiple faces without restricting the text prompts and layout of the diffusion model and without requiring manual intervention, without affecting the integrity of the faces, the diversity of the target images and the flexibility of the text prompts, thereby improving the accuracy of the generated images and being compatible with most diffusion models, achieving good scalability and universal compatibility.

[0206] See also Figure 8 , Figure 8 A flowchart of an information processing method based on a visual processing model provided according to an embodiment of the present specification is shown, which is applied to a task platform and specifically includes the following steps 802-804.

[0207] Step 802: Receive a model request sent by a terminal device, wherein the model request includes a scene identifier of a target scene, scene input data of the target scene, and at least one of a model specification parameter.

[0208] Step 804: Based on the model request, determine a corresponding target visual processing model from at least one visual processing model, wherein at least one visual processing model is trained based on the training method of the above-mentioned visual processing model.

[0209] It should be noted that based on the model request, the corresponding target visual processing model is determined from at least one visual processing model. One optional method is: based on the model request, the corresponding target visual processing model is searched from at least one visual processing model included in the model library; another optional method is: based on the model request, the target visual processing model is trained; and another optional method is: based on the model request, the target visual processing model is constructed, which is not limited here.

[0210] For example, based on the scene identification of the target scene, you can first search for at least one pre-trained visual processing model from the model library, then filter out a visual processing model of corresponding size from at least one visual processing model based on the model specification parameters, and then train the visual processing model of corresponding size based on the scene input data of the target scene to obtain a target visual processing model that meets user needs.

[0211] At least one visual processing model is obtained by training according to the training method of the visual processing model. Figure 7 The embodiments of the specification are based on the same inventive concept. For specific methods, please refer to the content of the above-mentioned model training, which will not be repeated here.

[0212] In an optional implementation of this embodiment, the model request includes a scene identifier of the target scene; based on the model request, determining a corresponding target visual processing model from at least one visual processing model includes:

[0213] Based on the scene identifier of the target scene, a target visual processing model adapted to the target scene is searched from a model library, wherein the model library stores at least one visual processing model adapted to different visual processing scenes.

[0214] It should be noted that the model library is a database for storing and managing various pre-trained deep learning models. Multiple visual processing models adapted to different visual processing scenarios cover different application scenarios and needs. The model library allows users to choose appropriate models according to their needs, or directly use the models for visual task processing through API calls.

[0215] The multiple visual processing models adapted to different visual processing scenarios are multiple models specially designed for different visual processing scenarios stored in the model library, each model is optimized for a specific application environment, and any visual processing model is obtained by training according to the visual processing model training method described above. For example, based on the scene identifier "image generation" of the target scene, the target visual processing model adapted to the image generation scene can be searched from the model library.

[0216] In the embodiments of this specification, based on the scene requirements, the target visual processing model suitable for the scene is accurately found through scene identification, so that the target processing result is more accurate and fits the scene, thereby improving the user experience and the quality of visual task processing.

[0217] As an example, the task platform can provide target visual processing models for a variety of scenarios, such as image generation scenarios, and can provide corresponding target visual processing models based on the model request sent by the terminal device. Since the target visual processing model is selected from at least one visual processing model trained by the training method based on the above-mentioned visual processing model, the target visual processing model has been precisely trained and can realize customized image generation of single objects and multiple objects.

[0218] In an optional implementation of this embodiment, the model request includes scene input data of the target scene; based on the model request, determining a corresponding target visual processing model from at least one visual processing model includes:

[0219] Determining an initial visual processing model adapted to the target scene from at least one visual processing model;

[0220] Based on the scene input data of the target scene, the initial visual processing model is trained to obtain the target visual processing model.

[0221] In actual implementation, the model request may include scene input data of the target scene, and the target visual processing model is a visual processing model suitable for the target scene.

[0222] Exemplarily, the general visual processing model is a basic visual processing model that is trained to be adaptable to different visual processing scenarios, but is not optimized for any specific scenario. For example, based on the scene input data of the image generation scenario, the general visual processing model is trained to obtain a target visual processing model adapted to the image generation scenario.

[0223] In the embodiments of this specification, based on scene requirements, the general visual processing model is further trained through scene input data to obtain a target visual processing model adapted to the scene, so that the target processing result is more accurate and fits the scene, thereby improving the user experience and the processing quality of visual tasks.

[0224] In an optional implementation of this embodiment, the model request includes model specification parameters; based on the model request, determining a corresponding target visual processing model from at least one visual processing model includes:

[0225] Based on the model specification parameters, a corresponding target visual processing model is searched from a model library, wherein the model library stores a plurality of visual processing models with different model specification parameters.

[0226] The model specification parameter may be a model size, such as searching for a target visual processing model of a corresponding size from a model library based on the model size: 32 GB.

[0227] In the embodiments of this specification, based on the model specification requirements, the corresponding target visual processing model is accurately found through the model specification parameters, which ensures the efficient and stable operation of the target visual processing model and improves the user experience.

[0228] In an optional implementation of this embodiment, after determining a corresponding target visual processing model from at least one visual processing model based on the model request, the method further includes:

[0229] Deploy the target visual processing model, and build a visual processing interface based on the target visual processing model so that the terminal device schedules the target visual processing model to perform visual tasks.

[0230] It should be noted that the visual processing interface is an interactive programming interface for the terminal device to schedule the target visual processing model, which is usually provided in the form of an API. Through the visual processing interface, the user can input the task data of the target visual task, such as at least one visual object, a reference image, and text prompt information, and effectively control the output of the model, such as the generated target image.

[0231] In actual implementation, an optional way to deploy the target visual processing model is to deploy the target visual processing model on the distributed system of the task platform. For example, the target visual processing model is deployed on the distributed system of the task platform, and based on the target visual processing model, a visual processing interface is constructed and provided to the terminal device, so that the terminal device schedules the target visual processing model to perform the target visual task of the visual processing scene.

[0232] In the embodiments of this specification, efficient terminal calling is achieved, visual task processing is optimized, and the processing quality and response speed of visual tasks are improved.

[0233] The information processing method based on the visual processing model provided in the embodiment of this specification is applied to obtain the target visual processing model according to user needs, realize personalized model service, provide users with an efficient, flexible and easy-to-use model service method, and improve user experience.

[0234] Corresponding to the above method embodiment, this specification also provides a task platform embodiment, Fig. 9 FIG. 1 shows a schematic diagram of a task platform provided by an embodiment of the present specification. Fig. 9 As shown, the task platform 900 includes: a request interface 902 and a response unit 904;

[0235] A request interface 902, used to receive a model request sent by a terminal device, wherein the model request includes a scene identifier of a target scene, scene input data of the target scene, and at least one of a model specification parameter;

[0236] The response unit 904 is used to determine the corresponding target visual processing model from at least one visual processing model based on the model request, wherein at least one visual processing model is trained based on the visual processing model training method as described above.

[0237] Optionally, the task platform further includes a visual processing interface, and the visual processing interface is constructed based on the target visual processing model;

[0238] Visual processing interface, used for terminal devices to schedule and execute visual tasks.

[0239] Optionally, the model request includes a scene identifier of the target scene;

[0240] The response unit 904 is further configured to:

[0241] Based on the scene identifier of the target scene, a target visual processing model adapted to the target scene is searched from a model library, wherein the model library stores at least one visual processing model adapted to different visual processing scenes.

[0242] Optionally, the model request includes scene input data of the target scene;

[0243] The response unit 904 is further configured to:

[0244] Determining an initial visual processing model adapted to the target scene from at least one visual processing model;

[0245] Based on the scene input data of the target scene, the initial visual processing model is trained to obtain the target visual processing model.

[0246] Optionally, the model request includes model specification parameters;

[0247] The response unit 904 is further configured to:

[0248] Based on the model specification parameters, a corresponding target visual processing model is searched from a model library, wherein the model library stores a plurality of visual processing models with different model specification parameters.

[0249] Optionally, the task platform further includes a deployment module configured to:

[0250] Deploy the target visual processing model, and build a visual processing interface based on the target visual processing model so that the terminal device schedules the target visual processing model to perform visual tasks.

[0251] In the embodiments of this specification, the task platform adapts to user needs to obtain the target visual processing model, realizes personalized model services, provides users with an efficient, flexible and easy-to-use model service platform, and improves the user experience.

[0252] The above is a schematic scheme of a task platform of this embodiment. It should be noted that the technical scheme of the task platform and the technical scheme of the information processing method based on the visual processing model belong to the same concept. For details not described in detail in the technical scheme of the task platform, please refer to the description of the technical scheme of the information processing method based on the visual processing model.

[0253] Corresponding to the above method embodiment, this specification also provides a visual task processing device embodiment, Fig.10 FIG. 1 is a schematic diagram showing the structure of a visual task processing device provided by an embodiment of the present specification. Fig.10 As shown, the device comprises:

[0254] A first acquisition module 1002 is configured to acquire task data of a target visual task, wherein the task data includes at least one visual object;

[0255] A first determination module 1004 is configured to extract an object embedding feature of at least one visual object, and determine spatial distribution information of at least one visual object according to the object embedding feature of the at least one visual object;

[0256] The acquisition module 1006 is configured to allocate at least one visual object to a corresponding object area using a target visual processing model according to spatial distribution information of at least one visual object, so as to obtain a target processing result of a target visual task.

[0257] Optionally, the first determining module 1004 is further configured to:

[0258] Obtaining object identity features of at least one visual object using an object recognition module of the embedding model;

[0259] Obtaining object structural features of at least one visual object using an object encoding module of the embedding model;

[0260] An object embedding feature of at least one visual object is obtained according to an object identity feature of at least one visual object and an object structure feature of at least one visual object by using a fusion module of the embedding model.

[0261] Optionally, the object recognition module includes a structure recognition network and an identity recognition network; the first determination module 1004 is further configured to:

[0262] Inputting at least one visual object into a structure recognition network to obtain a spatial structure representation of the at least one visual object;

[0263] The spatial structure representation of at least one visual object is input into an identity recognition network to obtain an object identity feature of the at least one visual object.

[0264] Optionally, the object encoding module includes an image encoder and a feature combination network; the first determination module 1004 is further configured to:

[0265] Sampling at least one visual object at different sizes to obtain structural representations of at least one visual object at different sizes;

[0266] Inputting at least one visual object into an image encoder to obtain an object encoding feature of the at least one visual object;

[0267] The object encoding features of at least one visual object and the structural representations of at least one visual object at different sizes are input into a feature combination network to obtain the object structural features of the at least one visual object.

[0268] Optionally, the fusion module includes a first attention mechanism layer, a second attention mechanism layer and a feature conversion layer; the first determination module 1004 is further configured as follows:

[0269] Inputting object identity features of at least one visual object into a first attention mechanism layer to obtain an identity feature representation sequence;

[0270] Inputting the identity feature representation sequence and the object structure feature of at least one visual object into the second attention mechanism layer to obtain the structure feature representation sequence;

[0271] The identity feature representation sequence and the structure feature representation sequence are input into the feature conversion layer to obtain the object embedding feature of at least one visual object.

[0272] Optionally, the first determining module 1004 is further configured to:

[0273] Performing attention calculation on the object embedding features of at least one visual object respectively to obtain an object representation corresponding to the at least one visual object;

[0274] Inputting the object embedding feature of at least one visual object into the object routing model to obtain an object position mask of at least one visual object, wherein one visual object corresponds to one object position mask;

[0275] Spatial distribution information of at least one visual object is obtained based on the object representation and the object position mask of the at least one visual object.

[0276] Optionally, the task data also includes reference images and / or text prompt information; the acquisition module 1006 is further configured to:

[0277] According to the reference image and / or text prompt information, and the spatial distribution information of at least one visual object, the target visual processing model is used to assign at least one visual object to the object area corresponding to the reference image to obtain the target processing result of the target visual task.

[0278] The visual task processing device provided in the embodiment of this specification includes a first acquisition module, a first determination module, and an acquisition module. Through the cooperation of the three functional modules, the target visual processing model is used to allocate at least one visual object to the corresponding object area, and the target processing result of the target visual task is obtained. At least one visual object is adaptively combined and distributed to the corresponding object area, and at least one visual object is allocated and managed to achieve customized generation of single visual objects and multiple visual objects without restricting the text prompts and layout of the target visual processing model, without requiring manual intervention, without affecting the integrity of the visual objects, the diversity of the target processing results and the flexibility of the text prompts, thereby improving the accuracy of visual task processing, and being compatible with most target visual processing models, achieving good scalability and universal compatibility.

[0279] The above is a schematic scheme of a visual task processing device of this embodiment. It should be noted that the technical scheme of the visual task processing device and the technical scheme of the visual task processing method described above belong to the same concept, and the details not described in detail in the technical scheme of the visual task processing device can be referred to the description of the technical scheme of the visual task processing method described above.

[0280] Corresponding to the above method embodiment, this specification also provides a visual task processing device embodiment, Fig.11 FIG. 1 is a schematic diagram showing a structure of a training device for a visual processing model provided by an embodiment of the present specification. Fig.11 As shown, the device comprises:

[0281] A second acquisition module 1102 is configured to acquire training data of a training task, wherein the training data includes at least one sample object and a corresponding result label;

[0282] The second determination module 1104 is configured to extract a sample object embedding feature of at least one sample object, and determine sample space distribution information of at least one sample object according to the sample object embedding feature of the at least one sample object;

[0283] The first training module 1106 is configured to train the initial visual processing model according to the sample space distribution information of at least one sample object in combination with the result label to obtain a trained target visual processing model.

[0284] Optionally, the training data includes first sample data and second sample data, and the second sample data includes at least two sample objects; the apparatus further includes a second training module configured to:

[0285] Using the first sample data, single-object training is performed on the initial embedding model and the initial visual processing model to obtain an updated visual processing model and a trained object embedding model;

[0286] Accordingly, the second determining module 1104 is further configured to:

[0287] Using the trained object embedding model, obtaining sample object embedding features of at least two sample objects;

[0288] The sample object embedding features of at least two sample objects are input into the initial routing model to obtain sample space distribution information of at least two sample objects.

[0289] Optionally, the second sample data further includes multi-object result labels; the first training module 1106 is configured to:

[0290] Obtaining a first prediction processing result using an initial visual processing model according to sample space distribution information of at least two sample objects;

[0291] According to the first prediction processing result and the multi-object result labels, the model loss is determined, and the model parameters of the initial routing model and the updated visual processing model are adjusted according to the model loss until the first training stop condition is reached, thereby obtaining the trained object routing model and the target visual processing model.

[0292] Optionally, the initial embedding model includes an initial recognition module, an initial encoding module, a first discarding module and an initial fusion module, the first sample data includes a single sample object and a corresponding single object result label; the second training module is further configured as follows:

[0293] Inputting a single sample object into an initial recognition module to obtain a sample object identity feature of the single sample object;

[0294] Using the initial encoding module, a sample object structure feature of a single sample object is obtained, and the sample object structure feature is input into a first discarding module to obtain an updated object structure feature, wherein the first discarding module is used to randomly discard part of the features in the sample object structure;

[0295] Input the sample object identity feature and the updated object structure feature into the initial fusion module to obtain the sample object embedding feature of the single sample object;

[0296] The sample object embedding feature of the single sample object is input into the initial visual processing model to obtain a second prediction result, and the model parameters of the initial embedding model and the initial visual processing model are adjusted according to the single object result label and the second prediction result until the second training stop condition is reached to complete the single object training.

[0297] Optionally, the initial fusion module includes a first initial attention mechanism layer, a second initial attention mechanism layer, a second discard module and an initial feature conversion layer; and the second training module is configured as follows:

[0298] Inputting the sample object identity feature into the first initial attention layer to obtain a sample identity feature representation sequence;

[0299] Input the sample identity feature representation sequence and the updated object structure feature into the second initial attention mechanism layer to obtain the sample structure feature representation sequence;

[0300] The sample structure feature representation sequence is input into the second discarding module, and the output result of the second discarding module is fused with the sample identity feature representation sequence to obtain the sample object embedding feature of the single sample object, wherein the second discarding module is used to discard the updated object structure feature based on the set probability.

[0301] Optionally, the first training module 1106 is configured to:

[0302] Performing attention calculations on sample object embedding features of at least two sample objects respectively to obtain object representations corresponding to the at least two sample objects;

[0303] Inputting sample object embedding features of at least two sample objects into an initial routing model to obtain sample position masks of at least two sample objects respectively, wherein one sample object corresponds to one sample position mask;

[0304] Obtaining sample spatial distribution information of at least two sample objects according to object representations and sample position masks of at least two sample objects;

[0305] Detecting the object boundaries of each sample object in the multi-object result labels;

[0306] Generate a label position mask for each sample object according to the object boundary of each sample object;

[0307] A routing loss is calculated according to the sample position mask and the label position mask, a prediction loss is calculated according to the first prediction processing result and the multi-object result labels, and a model loss is determined according to the routing loss and the prediction loss.

[0308] The training device of the visual processing model provided in the embodiment of this specification includes a second acquisition module 1102, a second determination module 1104, and a first training module 1106. Through the interaction of the three modules, the initial visual processing model can be trained according to the sample space distribution information of at least one sample object in combination with the result label to obtain a trained target visual processing model. Based on the sample space distribution information, information reference is provided for the initial visual processing model so that the initial visual processing model can learn to route at least one sample object and assign at least one sample visual object to a corresponding object area, so that the trained target visual model can realize customized generation of single visual objects and multiple visual objects without restricting the text prompts and layout of the target visual processing model.

[0309] The above is a schematic scheme of a visual processing model training device of this embodiment. It should be noted that the technical scheme of the visual processing model training device and the technical scheme of the visual processing model training method described above belong to the same concept, and the details not described in detail in the technical scheme of the visual processing model training device can be found in the description of the technical scheme of the visual processing model training method described above.

[0310] Fig.12 The block diagram of a computing device according to an embodiment of the present specification is shown. The components of the computing device 1200 include but are not limited to a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and the database 1250 is used to store data.

[0311] The computing device 1200 also includes an access device 1240, which enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0312] In one embodiment of the present specification, the above components of the computing device 1200 and Fig.12 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.12 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0313] The computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1200 may also be a mobile or stationary server.

[0314] Among them, the memory 1210 is used to store computer programs / instructions, and the processor 1220 is used to execute the computer programs / instructions stored in the memory 1210. When the computer program / instructions are executed by the processor, the steps of the above-mentioned visual task processing method, the training method of the visual processing model, or the information processing method based on the visual processing model are implemented.

[0315] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the visual task processing method, the training method of the visual processing model, or the information processing method based on the visual processing model belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the visual task processing method, the training method of the visual processing model, or the information processing method based on the visual processing model.

[0316] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual task processing method, visual processing model training method, or information processing method based on a visual processing model.

[0317] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the visual task processing method, the training method of the visual processing model, or the information processing method based on the visual processing model belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the visual task processing method, the training method of the visual processing model, or the information processing method based on the visual processing model.

[0318] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual task processing method, visual processing model training method, or information processing method based on a visual processing model.

[0319] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the visual task processing method, the training method of the visual processing model, or the information processing method based on the visual processing model belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the visual task processing method, the training method of the visual processing model, or the information processing method based on the visual processing model.

[0320] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0321] The computer program / instruction includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0322] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0323] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0324] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A visual task processing method, comprising: Acquire task data of a target visual task, wherein the task data includes at least one visual object; extracting an object embedding feature of the at least one visual object, and determining spatial distribution information of the at least one visual object according to the object embedding feature of the at least one visual object; According to the spatial distribution information of the at least one visual object, the at least one visual object is allocated to a corresponding object area using a target visual processing model to obtain a target processing result of the target visual task.

2. The method according to claim 1, wherein extracting the object embedding feature of the at least one visual object comprises: Obtaining an object identity feature of the at least one visual object using an object recognition module embedded in the model; Obtaining object structural features of the at least one visual object using an object encoding module of the embedding model; The object embedding feature of the at least one visual object is obtained according to the object identity feature of the at least one visual object and the object structure feature of the at least one visual object by using the fusion module of the embedding model.

3. The method according to claim 2, wherein the object recognition module comprises a structure recognition network and an identity recognition network; The obtaining of the object identity feature of the at least one visual object by using the object recognition module of the embedded model comprises: Inputting the at least one visual object into the structure recognition network to obtain a spatial structure representation of the at least one visual object; The spatial structure representation of the at least one visual object is input into the identity recognition network to obtain the object identity feature of the at least one visual object.

4. The method according to claim 2, wherein the object encoding module comprises an image encoder and a feature combination network; Before obtaining the object structure feature of the at least one visual object by using the object encoding module of the embedded model, the method further comprises: Sampling the at least one visual object at different sizes to obtain structural representations of the at least one visual object at different sizes; Accordingly, the object encoding module using the embedded model obtains the object structure feature of the at least one visual object, including: Inputting the at least one visual object into the image encoder to obtain an object coding feature of the at least one visual object; The object encoding features of the at least one visual object and the structural representations of the at least one visual object at different sizes are input into the feature combination network to obtain the object structural features of the at least one visual object.

5. The method according to claim 2, wherein the fusion module comprises a first attention mechanism layer, a second attention mechanism layer and a feature conversion layer; and the fusion module using the embedding model obtains the object embedding feature of the at least one visual object according to the object identity feature of the at least one visual object and the object structure feature of the at least one visual object, comprising: Inputting the object identity feature of the at least one visual object into the first attention mechanism layer to obtain an identity feature representation sequence; Inputting the identity feature representation sequence and the object structure feature of the at least one visual object into the second attention mechanism layer to obtain a structure feature representation sequence; The identity feature representation sequence and the structure feature representation sequence are input into the feature conversion layer to obtain the object embedding feature of the at least one visual object.

6. The method according to any one of claims 1 to 5, wherein determining the spatial distribution information of the at least one visual object according to the object embedding feature of the at least one visual object comprises: Performing attention calculations on the object embedding features of the at least one visual object respectively to obtain an object representation corresponding to the at least one visual object; Inputting the object embedding feature of the at least one visual object into the object routing model to obtain object position masks of the at least one visual object respectively, wherein one visual object corresponds to one object position mask; Spatial distribution information of the at least one visual object is obtained according to the object representation and the object position mask of the at least one visual object.

7. The method according to any one of claims 1 to 5, wherein the task data further includes a reference image and / or text prompt information; and the step of allocating the at least one visual object to a corresponding object area using a target visual processing model according to the spatial distribution information of the at least one visual object to obtain a target processing result of the target visual task comprises: According to the reference image and / or the text prompt information, and the spatial distribution information of the at least one visual object, the target visual processing model is used to assign the at least one visual object to the object area corresponding to the reference image to obtain the target processing result of the target visual task.

8. A method for training a visual processing model, comprising: Acquire training data for a training task, wherein the training data includes at least one sample object and a corresponding result label; extracting a sample object embedding feature of the at least one sample object, and determining sample space distribution information of the at least one sample object according to the sample object embedding feature of the at least one sample object; The initial visual processing model is trained according to the sample space distribution information of the at least one sample object in combination with the result label to obtain a trained target visual processing model.

9. The method according to claim 8, wherein the training data comprises first sample data and second sample data, wherein the second sample data comprises at least two sample objects; before extracting the sample object embedding feature of the at least one sample object, the method further comprises: Using the first sample data, single-object training is performed on the initial embedding model and the initial visual processing model to obtain an updated visual processing model and a trained object embedding model; Accordingly, extracting the sample object embedding feature of the at least one sample object and determining the sample space distribution information of the at least one sample object according to the sample object embedding feature of the at least one sample object includes: Using the trained object embedding model, obtaining sample object embedding features of the at least two sample objects; The sample object embedding features of the at least two sample objects are input into an initial routing model to obtain sample space distribution information of the at least two sample objects.

10. The method according to claim 9, wherein the second sample data further includes a multi-object result label; the training of the initial visual processing model based on the sample space distribution information of the at least one sample object and the result label to obtain a trained target visual processing model comprises: Obtaining a first prediction processing result using the updated visual processing model according to the sample space distribution information of the at least two sample objects; According to the first prediction processing result and the multi-object result labels, the model loss is determined, and the model parameters of the initial routing model and the updated visual processing model are adjusted according to the model loss until the first training stop condition is reached, thereby obtaining the trained object routing model and target visual processing model.

11. The method according to claim 9, wherein the initial embedding model comprises an initial recognition module, an initial encoding module, a first discarding module and an initial fusion module, and the first sample data comprises a single sample object and a corresponding single object result label; and the method of using the first sample data to perform single object training on the initial embedding model and the initial visual processing model to obtain an updated visual processing model and a trained object embedding model comprises: Inputting the single sample object into the initial recognition module to obtain the sample object identity feature of the single sample object; Using the initial encoding module, obtaining the sample object structure feature of the single sample object, and inputting the sample object structure feature into a first discarding module to obtain an updated object structure feature, wherein the first discarding module is used to randomly discard part of the features in the sample object structure; Inputting the sample object identity feature and the updated object structure feature into the initial fusion module to obtain the sample object embedding feature of the single sample object; The sample object embedding feature of the single sample object is input into the initial visual processing model to obtain a second prediction result, and the model parameters of the initial embedding model and the initial visual processing model are adjusted according to the single object result label and the second prediction result until the second training stop condition is reached, thereby completing the single object training.

12. The method according to claim 11, wherein the initial fusion module comprises a first initial attention mechanism layer, a second initial attention mechanism layer, a second discard module and an initial feature conversion layer; The step of inputting the sample object identity feature and the updated object structure feature into an initial fusion module to obtain the sample object embedding feature of the single sample object includes: Inputting the sample object identity feature into the first initial attention layer to obtain a sample identity feature representation sequence; Input the sample identity feature representation sequence and the updated object structure feature into the second initial attention mechanism layer to obtain a sample structure feature representation sequence; The sample structure feature representation sequence is input into a second discarding module, and the output result of the second discarding module is fused with the sample identity feature representation sequence to obtain the sample object embedding feature of the single sample object, wherein the second discarding module is used to discard the updated object structure feature based on a set probability.

13. The method according to claim 10, wherein the step of inputting the sample object embedding features of the at least two sample objects into the initial routing model to obtain the sample space distribution information of the at least two sample objects comprises: Performing attention calculations on the sample object embedding features of the at least two sample objects respectively to obtain object representations corresponding to the at least two sample objects; Inputting the sample object embedding features of the at least two sample objects into the initial routing model to obtain sample position masks of the at least two sample objects respectively, wherein one sample object corresponds to one sample position mask; Obtaining sample space distribution information of the at least two sample objects according to the object representations and the sample position masks of the at least two sample objects; Accordingly, determining the model loss according to the first prediction processing result and the multi-object result labels includes: Detecting the object boundary of each sample object in the multi-object result label; Generating a label position mask of each sample object according to an object boundary of each sample object; A routing loss is calculated according to the sample position mask and the label position mask, a prediction loss is calculated according to the first prediction processing result and the multi-object result labels, and the model loss is determined according to the routing loss and the prediction loss.

14. An information processing method based on a visual processing model, applied to a task platform, comprising: Receiving a model request sent by a terminal device, wherein the model request includes a scene identifier of a target scene, scene input data of the target scene, and at least one of a model specification parameter; Based on the model request, determining a corresponding target visual processing model from at least one visual processing model, wherein the at least one visual processing model is trained based on the visual processing model training method according to any one of claims 8 to 13; Wherein, after determining the corresponding target visual processing model from at least one visual processing model based on the model request, the method further includes: The target visual processing model is deployed, and a visual processing interface is constructed based on the target visual processing model so that the terminal device schedules the target visual processing model to perform visual tasks.

15. A task platform, comprising a request interface and a response unit; The request interface is used to receive a model request sent by a terminal device, wherein: The model request includes a scene identifier of a target scene, scene input data of the target scene, and at least one of a model specification parameter; The response unit is used to determine a corresponding target visual processing model from at least one visual processing model based on the model request, wherein the at least one visual processing model is trained based on the training method of the visual processing model as described in any one of claims 8-13.

16. The task platform according to claim 15, further comprising a visual processing interface, wherein the visual processing interface is constructed based on the target visual processing model; The visual processing interface is used for the terminal device to schedule and execute visual tasks.

17. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 14 are implemented.

18. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14.

19. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14.

Images