A model training method and related device

Abstract

The embodiment of the present application discloses a model training method and related devices in the field of artificial intelligence. The method obtains instance category features by performing semantic mapping processing on the decoding features generated when the image segmentation model processes the training images, and then mines the semantic conversion relationship between the instance category features corresponding to different training images, and accordingly performs semantic space conversion processing on the instance domain perception query corresponding to the semantic space of the instance category feature; because the instance domain perception query before and after the semantic space conversion processing should have semantic consistency, the target loss value for training the image segmentation model can be constructed according to the difference between the instance domain perception queries before and after the conversion, and the image segmentation model is trained based on the target loss value. The image segmentation model trained in the above manner can achieve good segmentation effect on each training sample set used.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method and related devices. Background Art

[0002] Image segmentation model is a common image processing model in the field of artificial intelligence. It is usually used to analyze and process input images to identify specific targets included in the input images and the category to which the specific targets belong.

[0003] To achieve better segmentation results for image segmentation models, the industry has proposed jointly training image segmentation models using multiple training sample sets. However, in practical applications, different training sample sets not only include different training images, but also typically correspond to different annotation systems. The annotation results under different annotation systems typically have different annotation accuracies and involve different annotation categories. Therefore, how to handle the differences between multiple training sample sets to achieve joint training of image segmentation models using multiple training sample sets has become a problem that such model training solutions need to solve.

[0004] At present, the implementation effects of the solutions proposed by relevant technologies for the above problems are generally not ideal, and the image segmentation models trained based on relevant technologies are usually difficult to achieve good segmentation effects. Summary of the Invention

[0005] The embodiments of the present application provide a model training method and related devices, which can improve the segmentation effect that can be achieved by an image segmentation model trained using multiple training sample sets.

[0006] In view of this, the first aspect of the present application provides a model training method, the method comprising:

[0007] Acquire a plurality of training images from a plurality of training sample sets, and acquire domain-aware queries corresponding to the plurality of training sample sets; the domain-aware queries correspond to the semantic space of the annotation system of the corresponding training sample sets;

[0008] The plurality of training images are respectively encoded by an encoder in the image segmentation model to obtain encoding features of the plurality of training images; the encoding features of the plurality of training images are respectively decoded by a decoder in the image segmentation model to obtain decoding features of the plurality of training images;

[0009] For each of the training images, semantic mapping is performed based on the decoded features of the training image to obtain instance category features corresponding to the training image; based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category features, an instance domain-aware query corresponding to the training image is determined; the instance domain-aware query corresponds to the semantic space of the instance category features;

[0010] Determining a semantic conversion relationship between instance category features corresponding to each of the plurality of training images; performing semantic space conversion processing on the instance domain-aware query corresponding to the training image based on the semantic conversion relationship, and determining a target loss value based on the instance domain-aware query before and after the conversion;

[0011] The image segmentation model is trained based on the target loss value.

[0012] A second aspect of the present application provides a model training device, the device comprising:

[0013] an acquisition module, configured to acquire a plurality of training images respectively from a plurality of training sample sets, and acquire a domain-aware query corresponding to each of the plurality of training sample sets; the domain-aware query corresponding to the semantic space of the annotation system of the corresponding training sample set;

[0014] an image processing module, configured to encode the plurality of training images using an encoder in the image segmentation model to obtain encoding features of the plurality of training images; and decode the encoding features of the plurality of training images using a decoder in the image segmentation model to obtain decoding features of the plurality of training images;

[0015] a feature processing module configured to perform semantic mapping on each of the training images based on the decoded features of the training images to obtain instance category features corresponding to the training images; determine an instance domain-aware query corresponding to the training image based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category features; the instance domain-aware query corresponds to the semantic space of the instance category features;

[0016] a loss value determination module, configured to determine a semantic conversion relationship between instance category features corresponding to each of the plurality of training images; based on the semantic conversion relationship, perform semantic space conversion processing on the instance domain-aware query corresponding to the training image, and determine a target loss value based on the instance domain-aware query before and after the conversion;

[0017] A model training module is used to train the image segmentation model based on the target loss value.

[0018] A third aspect of the present application provides a computer device, the device comprising a processor and a memory:

[0019] The memory is used to store computer programs;

[0020] The processor is used to execute the steps of the model training method described in the first aspect according to the computer program.

[0021] The fourth aspect of the present application provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the steps of the model training method described in the first aspect above.

[0022] In a fifth aspect, the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the model training method described in the first aspect.

[0023] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0024] The present invention provides a model training method that proposes a new training strategy for jointly training an image segmentation model using multiple training sample sets. Specifically, in the method, multiple training images are first obtained from multiple training sample sets, and domain-aware queries corresponding to each of the multiple training sample sets are obtained, where the domain-aware queries correspond to the semantic space of the annotation system of the corresponding training sample sets. Then, the multiple training images are sequentially encoded and decoded by an encoder and a decoder in the image segmentation model to obtain decoded features of each of the multiple training images. Then, for each training image, semantic mapping is performed on the decoded features of the training image to obtain instance category features corresponding to the training image. Based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category features, an instance domain-aware query corresponding to the training image is determined, where the instance domain-aware query corresponds to the semantic space of the instance category features. Furthermore, a semantic conversion relationship between the instance category features corresponding to each of the multiple training images is determined. Based on the semantic conversion relationship, a semantic space conversion process is performed on the instance domain-aware query corresponding to the training image. A target loss value is determined based on the instance domain-aware query before and after the conversion. Finally, the image segmentation model is trained using the target loss value. The above method performs semantic mapping on the decoding features generated when the image segmentation model processes the training image, and obtains the instance category features that reflect the mapping relationship between the decoding features and the annotation system of the training sample set to which the training image belongs. Then, the semantic conversion relationship between the instance category features corresponding to different training images is mined, and accordingly, the instance domain perception query corresponding to the semantic space of the instance category feature is subjected to semantic space conversion processing; considering that the instance domain perception query before and after the semantic space conversion processing should have semantic consistency, the embodiment of the present application accordingly constructs the target loss value for training the image segmentation model according to the difference between the instance domain perception queries before and after the conversion. The image segmentation model trained in the above manner can achieve good segmentation effect on each training sample set used. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of an application scenario of the model training method provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of the model training method provided in the embodiment of the present application;

[0027] Figure 3 A schematic diagram of the implementation architecture of the model training method provided in the embodiment of the present application;

[0028] Figure 4 A schematic diagram of the working principle of the intra-domain cross-attention module provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the working principle of the instance-aware cross-domain attention module provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of the working principle of the domain-aware cross-domain attention module provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of the model training device provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0033] Figure 9 A schematic diagram of the structure of the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0035] The terms "first," "second," "third," "fourth," etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatus.

[0036] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0037] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing them to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0038] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0039] The solutions provided in the embodiments of this application involve computer vision technology and machine learning technology in artificial intelligence technology, and are specifically described through the following embodiments:

[0040] In order to improve the performance of an image segmentation model trained using multiple training sample sets so that it can achieve better segmentation effects on multiple training sample sets, an embodiment of the present application provides a model training method.

[0041] Specifically, in the model training method provided in an embodiment of the present application, multiple training images are first obtained from multiple training sample sets, and domain-aware queries corresponding to each of the multiple training sample sets are obtained, where the domain-aware queries correspond to the semantic space of the annotation system of the corresponding training sample sets. Then, the multiple training images are encoded by an encoder in the image segmentation model to be trained to obtain encoded features for each of the multiple training images; and the encoded features of each of the multiple training images are decoded by a decoder in the image segmentation model to obtain decoded features for each of the multiple training images. Next, for each training image, semantic mapping is performed based on the decoded features of the training image to obtain instance category features corresponding to the training image. Furthermore, based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category features, an instance domain-aware query corresponding to the training image is determined, where the instance domain-aware query corresponds to the semantic space of the instance category features. Furthermore, a semantic conversion relationship between the instance category features corresponding to each of the multiple training images is determined, and based on the semantic conversion relationship, a semantic space conversion is performed on the instance domain-aware query corresponding to the training image, and a target loss value is determined based on the instance domain-aware query before and after the conversion. Finally, the above image segmentation model is trained based on the target loss value.

[0042] The above-mentioned model training method performs semantic mapping processing on the decoding features generated when the image segmentation model to be trained processes the training image, and obtains the instance category features that reflect the mapping relationship between the decoding features and the labeled categories involved in the training sample set to which the training image belongs, and then mines the semantic conversion relationship between the instance category features corresponding to different training images, and accordingly performs semantic space conversion processing on the instance domain perception query corresponding to the semantic space of the instance category feature. Taking into account that the instance domain perception query before and after the semantic space conversion processing should have semantic consistency, the embodiment of the present application accordingly constructs a target loss value for training the image segmentation model according to the difference between the instance domain perception queries before and after the conversion, and trains the image segmentation model based on the target loss value. The image segmentation model trained in the above manner can achieve good segmentation effects on each training sample set used.

[0043] It should be understood that the model training method provided in the embodiments of the present application can be executed by a computer device with model training capabilities, which can be a terminal device or a server. Among them, terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server. In addition, the relevant data involved in the embodiments of the present application can be stored in the blockchain network.

[0044] In order to facilitate understanding of the model training method provided in the embodiment of the present application, the application scenario of the model training method is exemplarily introduced below, taking the execution subject of the model training method as a server as an example.

[0045] See also Figure 1 , Figure 1 Schematic diagram of the application scenario of the model training method provided in the embodiment of this application. Figure 1 As shown, the application scenario includes a server 110, a database 120, and a database 130; the server 110 can access the database 120 and the database 130 through a network, or the database 120 and the database 130 can also be integrated into the server 110. The server 110 is used to execute the model training method provided in the embodiment of the present application to train the image segmentation model using different training sample sets; the database 120 and the database 130 store different training sample sets, each of which includes a large number of training samples, and each training sample includes a training image and its corresponding labeled category.

[0046] In actual applications, the server 110 can obtain training images from the training sample sets stored in the database 120 and the database 130 respectively; for example, assuming that the database 120 is used to store the training sample set A and the database 130 is used to store the training sample set B, then the server 110 can obtain the training image a from the training sample set A stored in the database 120 and obtain the training image b from the training sample set B stored in the database 130; it should be understood that in actual applications, the server 110 can also obtain more training images from more training sample sets, and this application does not impose any limitation on the number of training sample sets used by the server 110 when training the image segmentation model.

[0047] In addition, server 110 also needs to obtain the domain-aware query corresponding to the training sample set to which each training image belongs. For example, assuming that server 110 obtains training image a and training image b when training the image segmentation model, server 110 needs to obtain the domain-aware query corresponding to training sample set A to which training image a belongs, and the domain-aware query corresponding to training sample set B to which training image b belongs. The domain-aware query here corresponds to the semantic space of the annotation system of the corresponding training sample set. Specifically, each training sample set has its own corresponding annotation system, which is determined by the training images included in the training sample set and the annotation categories corresponding to the training images. The annotation system has its own corresponding semantic space. The domain-aware query corresponding to the training sample set is used to reflect the characteristics of the semantic space of the annotation system corresponding to the training sample set.

[0048] When specifically training the image segmentation model, the server 110 can first encode the multiple training images obtained through the encoder in the image segmentation model to obtain the encoding features of each of the multiple training images; then, use the decoder in the image segmentation model to decode the encoding features of each of the multiple training images to obtain the decoding features of each of the multiple training images.

[0049] For each training image, server 110 can perform semantic mapping on the decoded features of the training image to obtain the instance category features corresponding to the training image. The instance category features can reflect the mapping relationship between the decoded features and the annotation system corresponding to the training sample set to which the training image belongs. Server 110 can then determine the instance domain-aware query corresponding to the training image based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category features. The instance domain-aware query corresponds to the semantic space of the instance category features.

[0050] Furthermore, server 110 can mine the semantic conversion relationships between instance category features corresponding to each of the multiple training images and, based on these semantic conversion relationships, perform semantic space conversion processing on the instance domain-aware query corresponding to the training image, converting the instance domain-aware query corresponding to the training image to the semantic space corresponding to the instance domain-aware query corresponding to another training image. Because the instance domain-aware query before and after the semantic space conversion processing should be semantically consistent, server 110 can determine a target loss value for training the image segmentation model based on the difference between the instance domain-aware queries before and after the conversion, and train the image segmentation model based on this target loss value.

[0051] It should be understood that Figure 1 The application scenarios shown are merely examples. In actual applications, the model training methods provided in the embodiments of the present application can also be applied to other scenarios. For example, the server 110 can obtain training images used to train the image segmentation model from more or fewer databases. For another example, the server 110 can also obtain training images used to train the image segmentation model through other channels. In addition, the model training methods provided in the embodiments of the present application can be applied to any scenario requiring the application of an image segmentation model, such as gaming scenarios and traffic scenarios. The application scenarios of the model training methods provided in the embodiments of the present application are not limited herein.

[0052] The model training method provided in this application is introduced in detail below through a method embodiment.

[0053] See also Figure 2 , Figure 2The flow chart of the model training method provided in the embodiment of the present application is as follows. For the sake of convenience, the following embodiment is still introduced by taking the execution subject of the model training method as the server as an example. Figure 2 As shown, the model training method includes the following steps:

[0054] Step 201: Acquire multiple training images from multiple training sample sets, and acquire domain-aware queries corresponding to the multiple training sample sets; the domain-aware queries correspond to the semantic space of the annotation system of the corresponding training sample sets.

[0055] In an embodiment of the present application, when a server jointly trains an image segmentation model using multiple training sample sets, it is necessary to obtain training images from each of the multiple training sample sets so that the image segmentation model to be trained can be used to process each of the training images obtained from the multiple training sample sets. The multiple training sample sets refer to at least two training sample sets, and accordingly, the multiple training images also refer to at least two training images.

[0056] For example, assuming that the server needs to use training sample set A and training sample set B to jointly train the image segmentation model, the server can obtain training image a from training sample set A and training image b from training sample set B. When training the image segmentation model, the server can use the image segmentation model to be trained to process training image a and training image b respectively.

[0057] It should be understood that in actual applications, the server can use more training sample sets to train the image segmentation model according to actual needs. This application does not impose any restrictions on the number of training sample sets used when training the image segmentation model. Accordingly, the server can also obtain the training images required to be processed by the image segmentation model to be trained from more training sample sets. This application does not impose any restrictions on the number of training images used each time the image segmentation model is trained.

[0058] It should be noted that the image segmentation model to be trained in the embodiment of the present application is a neural network model for performing image segmentation tasks, which is used to identify the specific target included in the input image and the category to which the specific target belongs. Exemplarily, the image segmentation model can be a human body segmentation model, which is used to locate the various regions including human body parts in the input image, and identify the human body parts corresponding to each region; for example, the head region, limb region, torso region, etc. of the person included in the input image are identified; of course, the image segmentation model can also be a model for performing other segmentation tasks, and the present application does not make any limitation on the segmentation tasks performed by the image segmentation model to be trained. The structure of the image segmentation model to be trained in the embodiment of the present application can be, for example, a U-Net structure, or more specifically, an encoder-decoder structure built based on the Transformer module. Of course, the image segmentation model can also be expressed as other structures, and the present application does not make any limitation on the structure of the image segmentation model.

[0059] In an embodiment of the present application, the multiple training sample sets used to train the image segmentation model are all training sample sets related to the segmentation task of the image segmentation model. For example, assuming that the image segmentation model to be trained is a human body segmentation model, each training sample set used to train the human body segmentation model is a training sample set related to the human body segmentation task. Specifically, the training images in each training sample set need to include a person, and the annotation result corresponding to each training image is the area in the training image that includes a human body part, and the specific human body part corresponding to the area.

[0060] In practical applications, different training sample sets correspond to different annotation systems, and different annotation systems usually correspond to different annotation accuracies, and the annotation categories involved are usually different. Taking the training sample set used to train the human segmentation model as an example, the annotation results corresponding to the training images in some training sample sets may have lower annotation accuracy. For example, the annotation results corresponding to the training images can only reflect the head area, torso area, and limbs area of the person in the training image; while the annotation results corresponding to the training images in other training sample sets may have higher annotation accuracy. For example, the annotation results corresponding to the training images can reflect in detail the areas where the facial features of the person in the training image are located, and the areas where the main body parts (such as the chest, waist, abdomen, buttocks, etc.) are located.

[0061] In an embodiment of the present application, when a server jointly trains an image segmentation model using multiple training sample sets, in addition to obtaining training images from each of the multiple training sample sets, it also needs to obtain domain-aware queries corresponding to each of the multiple training sample sets. For example, assuming that the server needs to jointly train an image segmentation model using training sample set A and training sample set B, the server needs to obtain the domain-aware query corresponding to training sample set A and the domain-aware query corresponding to training sample set B.

[0062] It should be noted that the domain-aware query (DSQ) corresponding to the training sample set corresponds to the semantic space of the annotation system of the training sample set. In other words, the domain-aware query corresponding to the training sample set can also be understood as a feature vector used to characterize the semantic space of the annotation system of the training sample set. Specifically, the annotation system of the training sample set is usually determined by the features of the training images included in the training sample set and the annotation results corresponding to the training images. The annotation system can constitute a corresponding semantic domain (Semantic Domain), which is also the semantic space corresponding to the training sample set. The domain-aware query corresponding to the training sample set is used to reflect the features of the semantic space corresponding to the training sample set. Exemplarily, the domain-aware query corresponding to the training sample set can be expressed as Among them, N is the number of annotation categories involved in the training sample set, and L is the length of the feature vector corresponding to each annotation category; in practical applications, the domain-aware query corresponding to the training sample set can be learned based on the features of each training image included in the training sample set and the annotation results corresponding to each training image.

[0063] Step 202: Encode the plurality of training images respectively through the encoder in the image segmentation model to obtain encoding features of the plurality of training images; and decode the encoding features of the plurality of training images respectively through the decoder in the image segmentation model to obtain decoding features of the plurality of training images.

[0064] After the server obtains multiple training images from different training sample sets, it can further process these multiple training images using the image segmentation model to be trained. Specifically, for each training image, the server can encode the training image using the encoder in the image segmentation model to obtain the encoded features of the training image. Then, the server can decode the encoded features of the training image using the decoder in the image segmentation model to obtain the decoded features of the training image.

[0065] It should be noted that in practical applications, image segmentation models typically include multiple cascaded encoders and decoders. The encoder encodes the original input image features or the encoded features output by the previous encoder, i.e., it reduces the resolution of the input feature map and doubles the number of channels. The decoder decodes the encoded features output by the last encoder or the decoded features output by the previous decoder, i.e., it increases the resolution of the input feature map and halves the number of channels.

[0066] For example, see Figure 3 , Figure 3 A schematic diagram of the implementation architecture of the model training method provided in an embodiment of the present application. Figure 3 Take the trained image segmentation model as an encoder-decoder architecture built using the Transformer module, and use two training sample sets to jointly train the image segmentation model as an example, Figure 3 As shown, the input of each iteration is from two training sample sets {D s ,D t} training images In the encoding stage, the Transformer module and downsampling operation are used to halve the resolution of the feature map and double the number of channels. After N downsampling, N+1 sets of encoding features of different feature scales will be obtained. The encoding feature corresponding to feature scale i can be expressed as In the decoding stage, starting from the encoding feature with the smallest resolution, the Transformer module and upsampling operation are used to increase the resolution of the feature map and halve the number of channels; after the decoder outputs the decoded feature, the decoded feature can be added to the encoding feature of the same feature scale as the input feature of the next decoder.

[0067] It should be understood that when the image segmentation model includes N decoders, the server executes step 202 to obtain N decoding features for each training image, and these N decoding features have different feature scales.

[0068] Step 203: For each of the training images, semantic mapping is performed based on the decoding features of the training image to obtain the instance category features corresponding to the training image; based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category features, the instance domain-aware query corresponding to the training image is determined; the instance domain-aware query corresponds to the semantic space of the instance category features.

[0069] After the server processes multiple training images using an image segmentation model and obtains the decoded features of each of the training images, it can perform semantic mapping on each training image based on the decoded features of the training image to obtain the instance category feature corresponding to the training image. The instance category feature is a feature that reflects the mapping relationship between the decoded features and the annotation system corresponding to the training sample set to which the training image belongs. In other words, the instance category feature is a feature obtained by performing category mapping based on the decoded features and can, to a certain extent, reflect the category prediction result of the training image under the annotation system of the training sample set to which the training image belongs.

[0070] Furthermore, for each training image, the server can also determine the instance domain-aware query corresponding to the training image based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category feature corresponding to the training image. The instance domain-aware query corresponds to the semantic space of the instance category feature and can also be understood as a feature vector used to represent the semantic space of the corresponding instance category feature.

[0071] In one possible implementation, the server can distinguish between different feature scales and determine, for each training image, the instance category features and instance domain-aware queries corresponding to each feature scale. Specifically, when the decoded features of each training image include decoded features of N (N is an integer greater than 1) feature scales, when determining the instance category features and instance domain-aware queries corresponding to each training image, semantic mapping can be performed on each feature scale based on the decoded features of the feature scale to obtain the instance category features corresponding to the feature scale. Furthermore, based on the domain-aware queries corresponding to the training sample set to which the training image belongs and the instance category features corresponding to each of the N feature scales of the training image, instance domain-aware queries corresponding to each of the N feature scales are determined. Here, there is a correspondence between the instance category features corresponding to the same feature scale and the instance domain-aware queries.

[0072] Determining the instance category features corresponding to a training image is essentially generating a semantic mapping of the decoding features in the semantic space of the training sample set to which the training image belongs. For example, for a training image The corresponding decoding feature with feature scale λ can be expressed as When performing semantic mapping based on the decoding features, we can first use the mapping matrix Calculate the weight of the decoded feature for each category (each labeled category involved in the training sample set to which the training image belongs), where N is the total number of labeled categories involved in the training sample set; then, calculate the reference instance category feature corresponding to the feature scale λ based on the weight; and then, use the transformation matrix Convert the feature dimensions corresponding to each category in the reference instance category feature to d, and obtain the instance category feature corresponding to the feature scale λ of the training image I The above process of determining the instance category feature corresponding to the feature scale λ can be specifically expressed as the following formula (1):

[0073]

[0074] Furthermore, the server can determine the instance domain-aware queries corresponding to each of the N feature scales based on the domain-aware queries corresponding to the training sample set to which the training image belongs and the instance category features of the training image corresponding to the N feature scales; the instance domain-aware queries corresponding to each feature scale are used to represent the semantic space of the instance category features corresponding to the feature scale, that is, there is a corresponding relationship between the instance domain-aware queries and instance category features corresponding to the same feature scale.

[0075] As an example, the server may determine the instance domain-aware query corresponding to each feature scale in the following manner: for each feature scale, based on the reference instance domain-aware query corresponding to the feature scale and the instance category features corresponding to the feature scale, determine the instance domain-aware query corresponding to the feature scale. The reference instance domain-aware query corresponding to the smallest feature scale among the N feature scales is the domain-aware query corresponding to the training sample set to which the training image belongs; and the reference instance domain-aware query corresponding to each feature scale among the N feature scales, except for the smallest feature scale, is the instance domain-aware query corresponding to a feature scale among the N feature scales that is smaller than and adjacent to the feature scale.

[0076] like Figure 3 As shown, for the minimum feature scale among the N feature scales, the server can use the domain-aware query corresponding to the training sample set to which the training image belongs as the reference instance domain-aware query corresponding to the minimum feature scale; then, based on the reference instance domain-aware query and the instance category features corresponding to the minimum feature scale, the instance domain-aware query corresponding to the minimum feature scale is obtained. The instance domain-aware query corresponding to the minimum feature scale will be further used as the reference instance domain-aware query corresponding to the next feature scale. Here, the next feature scale is the feature scale that is larger than the minimum feature scale and closest to the minimum feature scale among the N feature scales. The server can interact with the instance category features corresponding to the feature scale based on the reference instance domain-aware query to obtain the instance domain-aware query corresponding to the feature scale. This process can be deduced in this way until the instance domain-aware query corresponding to the maximum feature scale among the N feature scales is obtained.

[0077] Through the above processing, the server can obtain the instance domain-aware queries corresponding to each of the N feature scales, namely {Q1, Q2, ..., Q N}, Q1, Q2, ..., Q N The corresponding feature scales are arranged from large to small. In order to unify the representation, the domain-aware query corresponding to the training sample set to which the training image belongs can be denoted as Q N+1 , that is, the domain-aware query corresponding to the training sample set is regarded as the instance domain-aware query corresponding to the N+1th feature scale, then the N+1 instance domain-aware query can be expressed as

[0078] When the server specifically determines the instance domain-aware query corresponding to each feature scale, it can first determine the semantic association relationship between the various annotation categories involved in the training image; then, the reference instance domain-aware query corresponding to this feature scale is used as the query element in the attention mechanism, and the instance category features of this feature scale are used as the key element and value element in the attention mechanism; then, based on the attention mechanism, according to the above query element, key element and value element, as well as the semantic association relationship, the instance domain-aware query corresponding to this feature scale is determined.

[0079] It should be noted that the query elements, key elements, and value elements in the attention mechanism are similar to those in the retrieval scenario. For example, when a user searches for specific content through a search engine, they enter the search keyword in the search bar (the search keyword is equivalent to the query element). The backend then maps the search keyword to the title, summary, and other descriptive information related to the candidate search content in the database (the descriptive information is equivalent to the key element). Furthermore, the backend then feeds back the best matching result to the user (the matching result is equivalent to the value element). In other words, the attention mechanism can also be understood as a retrieval process. In the attention mechanism, based on the query elements, key elements, and value elements mentioned above, features with higher importance and greater ability to attract attention can be extracted from the input features.

[0080] Taking the trained image segmentation model as a human body segmentation model as an example, when the server determines the semantic association relationship between the various annotation categories involved in the training image, it can be determined based on the positional relationship between the human body parts corresponding to each annotation category. For example, the server can construct an adjacency matrix to represent the semantic association relationship. Among them, N is the total number of labeled categories involved in the training image. In this adjacency matrix, a i,j =1 means that the human body parts corresponding to the i-th and j-th annotation categories are adjacent, a i,j =0 means that the human body parts corresponding to the i-th labeling category and the j-th labeling category are not adjacent.

[0081] like Figure 3 As shown in Figure 2, for the feature scale λ, its corresponding reference instance domain-aware query can be interacted with the instance category features through the intra-domain cross-attention module to obtain the instance domain-aware query corresponding to the feature scale λ.

[0082] See also Figure 4 , Figure 4 This is a schematic diagram of the working principle of the intra-domain cross attention module provided in the embodiment of the present application. Figure 4 As shown, for the feature scale λ, the instance domain-aware query Q corresponding to the previous feature scale can be λ+1 (i.e., reference instance domain-aware query) is used as the query element (Query) in the multi-head cross attention mechanism, and the instance category feature Z corresponding to the feature scale λ is used. I,λ As the key element (Key) and value element (Value) in the multi-head cross attention mechanism. Furthermore, based on the multi-head cross attention mechanism, the instance domain perception query corresponding to the feature scale λ can be determined according to the query elements, key elements and value elements, as well as the semantic association relationship; specifically, the adjacency rejection adjacency matrix A used to represent the semantic association relationship can be Intra As a mask, it acts on the domain-aware query Q based on the reference instance λ+1 and instance category feature Z I,λ The generated attention map embeds the semantic prior knowledge between the labeled categories; then, the features fused with instance information are further processed by layer regularization and feedforward neural network, and the processing results are consistent with the reference instance domain perception query Q λ+1 Add together to generate the instance domain-aware query Q corresponding to the feature scale λ λ .

[0083] Specifically, the output feature of the j-th head in the multi-head cross attention mechanism can be expressed as formula (2):

[0084]

[0085] Among them, W q,j 、W k,j and W v,j is the model parameter associated with the jth head in the multi-head cross attention mechanism, d is the dimension of the feature vector in the instance domain aware query. · represents the matrix inner product, Represents pixel-level multiplication.

[0086] Then, the output features of each head in the multi-head cross attention mechanism are spliced through formula (3) to obtain the output result of the multi-head cross attention mechanism:

[0087] IntraMHCA(Q λ+1 ,Z I,λ ,Aintra )=concat(IntraCA1,…,IntraCA M ) (3)

[0088] Then, the output of the multi-head cross attention mechanism is added to the reference instance domain-aware query to obtain Q′ λ , and Q′ is obtained through layer regularization and feedforward network λ Processing is performed, and finally the processing results are compared with Q′ λ Add together to get the instance domain-aware query Q corresponding to the feature scale λ The processing can be expressed as the following equations (4) and (5):

[0089] Q′ λ =IntraMHCA(Q λ+1 ,Z I,λ ,A intra )+Q λ+1 (4)

[0090] Q λ =FFN(Norm(Q′ λ ))+Q′ λ (5)

[0091] In this way, by determining the instance category features and instance domain-aware queries corresponding to each feature scale in the above manner, it can be ensured that the determined instance category features more accurately reflect the mapping relationship between the decoding features and the annotation system, and that the determined instance domain-aware queries more accurately reflect the semantic space of the instance category features, thereby facilitating the subsequent accurate determination of the loss value used in training the model.

[0092] Step 204: Determine the semantic conversion relationship between the instance category features corresponding to each of the multiple training images; based on the semantic conversion relationship, perform semantic space conversion processing on the instance domain-aware query corresponding to the training image, and determine the target loss value according to the instance domain-aware query before and after the conversion.

[0093] After determining the instance category features and instance domain-aware queries corresponding to each training image in step 203, the server can further determine a semantic conversion relationship between the instance category features corresponding to the multiple training images based on the instance category features corresponding to the multiple training images. The semantic conversion relationship here can be used to implement feature conversion between semantic spaces corresponding to different instance category features. For example, based on the semantic conversion relationship determined between instance category feature x and instance category feature y, features in the semantic space corresponding to instance category feature x can be converted to the semantic space corresponding to instance category feature y, and features in the semantic space corresponding to instance category feature y can be converted to the semantic space corresponding to instance category feature x.

[0094] After the server determines the semantic conversion relationship between the instance category features corresponding to each of the multiple training images, it can perform semantic space conversion processing on the instance domain perception query corresponding to the training image related to the semantic conversion relationship based on the semantic conversion relationship. The so-called training image related to the semantic conversion relationship refers to the training image to which the semantic conversion relationship can be applied. For example, assuming that a certain semantic conversion relationship is determined based on the instance category features corresponding to the training image a and the training image b, then the training image a and the training image b both belong to the training images related to the semantic conversion relationship, while the training image c does not belong to the training image related to the semantic conversion relationship. The so-called semantic space conversion processing refers to the processing method of converting the instance domain perception query from the semantic space to which it originally belongs to to another semantic space.

[0095] Because different features should maintain semantic consistency even within different semantic spaces, the server can use the difference between the converted instance-domain-aware query and the unconverted instance-domain-aware query as the target loss for model training. Specifically, the server performs semantic space conversion on the instance-domain-aware query corresponding to a training image. After obtaining the converted instance-domain-aware query, the server can calculate the difference between the converted instance-domain-aware query and the unconverted instance-domain-aware query as the target loss for model training.

[0096] In one possible implementation, when the decoding features of the training images generated by the image segmentation model include decoding features of N feature scales, the server can generate instance category features and instance domain-aware queries corresponding to each feature scale for each training image. In this case, for the target feature scale, the server can determine the semantic conversion relationship corresponding to the target feature scale based on the instance category features corresponding to the target feature scale of multiple training images; then, based on the semantic conversion relationship corresponding to the target feature scale, the instance domain-aware queries corresponding to the target feature scale of the training images are subjected to semantic space conversion processing, and the target loss value corresponding to the target feature scale is determined based on the instance domain-aware queries before and after the conversion.

[0097] It should be understood that the target characteristic scale can be any one of the N characteristic scales. Figure 3 As shown, for each feature scale, the server can determine the target loss value corresponding to the feature scale based on the instance category features of multiple training images corresponding to the feature scale and the instance domain-aware query through the instance-aware cross-domain attention module. Figure 3 As shown, for each feature scale, the server can determine a target loss value corresponding to the feature scale for each training image, that is, the server can perform semantic space conversion processing on the instance domain perception query corresponding to each training image based on the determined semantic conversion relationship, and then construct the target loss value corresponding to the feature scale of the training image according to the instance domain perception query before and after the conversion corresponding to each training image.

[0098] As an example, the server can specifically determine the target loss value corresponding to the target feature scale in the following manner: based on the instance domain-aware queries corresponding to the target feature scale of each of the multiple training images, obtain at least one instance domain-aware query combination, each instance domain-aware query combination including two instance domain-aware queries; for each instance domain-aware query combination, determine one of the instance domain-aware queries as the source instance domain-aware query, and determine the other instance domain-aware query as the target instance domain-aware query; then, determine the semantic conversion relationship based on the instance category features corresponding to the source instance domain-aware query and the target instance domain-aware query; and based on the semantic conversion relationship, convert the source instance domain-aware query to the semantic space corresponding to the target instance domain-aware query to obtain a converted instance domain-aware query; finally, determine the target loss value corresponding to the instance domain-aware query combination based on the source instance domain-aware query and the converted instance domain-aware query.

[0099] Specifically, the server may combine instance domain-aware queries corresponding to target feature scales of the plurality of training images in pairs to obtain at least one instance domain-aware query combination, where each instance domain-aware query combination includes two instance domain-aware queries.

[0100] For including and Instance domain aware query combination (where, is the instance domain aware query corresponding to the feature scale λ for the training images from the training sample set s, is the instance domain-aware query corresponding to the feature scale λ of the training image from the training sample set t), the server can As the source instance domain-aware query, As the target instance domain-aware query, and based on the The corresponding instance category features and the The corresponding instance category features determine the semantic conversion relationship. For example, the adjacency matrix can be calculated by the following formula (6): As the semantic conversion relationship:

[0101]

[0102] in, and They are and The corresponding instance category features, W s,λ and W t,λ is the feature conversion parameter.

[0103] Furthermore, based on the semantic conversion relationship determined in the above manner, the source instance domain-aware query can be converted to the semantic space corresponding to the target instance domain-aware query to obtain a converted instance domain-aware query; the converted instance domain-aware query and the source instance domain-aware query should have semantic consistency, so the target loss value corresponding to the instance domain-aware query combination can be determined based on the difference between the converted instance domain-aware query and the source instance domain-aware query. It should be understood that in actual applications, the source instance domain-aware query and the target instance domain-aware query can also be swapped, that is, the target instance domain-aware query is converted to the semantic space corresponding to the source instance domain-aware query to obtain a converted instance domain-aware query, and based on the difference between the converted instance domain-aware query and the target instance domain-aware query, another target loss value corresponding to the instance domain-aware query combination is determined.

[0104] As an example, the server can specifically convert the source instance domain-aware query to the semantic space corresponding to the target instance domain-aware query in the following manner to obtain a converted instance domain-aware query: use the target instance domain-aware query as the query element in the attention mechanism, and use the source instance domain-aware query as the key element and value element in the attention mechanism; based on the attention mechanism, according to the above-mentioned query elements, key elements and value elements, and semantic conversion relationships, convert the source instance domain-aware query to the semantic space corresponding to the target instance domain-aware query to obtain the converted instance domain-aware query.

[0105] See also Figure 5 , Figure 5 This is a schematic diagram of the working principle of the instance-aware cross-domain attention module provided in the embodiment of this application. Figure 5 As shown, in the instance-aware cross-domain attention module, the target instance domain-aware query As the query element in the multi-head cross attention mechanism, the source instance domain-aware query As the key element and value element in the multi-head cross attention mechanism, the and The semantic conversion relationship B determined by the corresponding instance category features local As a mask, based on and The generated attention map completes the semantic space transformation through the cross-attention mechanism.

[0106] Specifically, the output feature of the j-th head in the multi-head cross attention mechanism can be expressed as formula (7):

[0107]

[0108] Among them, W q,j 、W k,j and W v,j is the model parameter associated with the jth head in the multi-head cross attention mechanism, and d is the dimension of the feature vector in the instance domain aware query.

[0109] Then, the output features of each head in the multi-head cross attention mechanism are spliced together by formula (8) to obtain the output result of the multi-head cross attention mechanism:

[0110]

[0111] Then, the output of the multi-head cross attention mechanism is combined with the target instance domain perception query Add them together to get Q′ λ , and Q′ is obtained through layer regularization and feedforward network λ Processing is performed, and finally the processing results are compared with Q′ λAdd together to get the transformed instance domain-aware query The processing can be expressed as follows:

[0112]

[0113]

[0114] Finally, we can use the following formula (11) to query the source instance domain perception Its corresponding transformation instance domain-aware query Determine the target loss value L icr :

[0115]

[0116] In this way, by determining the target loss value in the above manner, it is possible to ensure accurate semantic space conversion processing of instance domain-aware queries, thereby ensuring the accuracy of the target loss value determined based on the instance domain-aware queries before and after the conversion, which is conducive to ensuring the reliability of model training.

[0117] In one possible implementation, in addition to performing semantic space conversion on instance domain-aware queries and using loss values for constructing training models based on the pre- and post-conversion instance domain-aware queries, the server can also perform semantic space conversion on domain-aware queries corresponding to training sample sets and use loss values for constructing training models based on the pre- and post-conversion domain-aware queries. Specifically, the server can determine the category mapping relationships between the annotation systems of multiple training sample sets; based on these category mapping relationships, it can then perform semantic space conversion on the domain-aware queries corresponding to the training sample sets and determine reference loss values based on the pre- and post-conversion domain-aware queries.

[0118] Specifically, the server can determine the category mapping relationship between the annotation systems of multiple training sample sets based on the annotation categories involved in each of the multiple training sample sets. The category mapping relationship here can be used to realize the conversion of features between the semantic spaces of the annotation systems of different training sample sets. For example, based on the category mapping relationship determined based on the annotation categories involved in training sample set A and the annotation categories involved in training sample set B, the features in the semantic space corresponding to the annotation system of training sample set A can be converted to the semantic space corresponding to the annotation system of training sample set B.

[0119] After the server determines the category mapping relationship between the annotation systems of multiple training sample sets, it can perform semantic space conversion processing on the domain-aware queries corresponding to the training sample sets related to the category mapping relationship based on the category mapping relationship. The so-called training sample sets related to the category mapping relationship refer to the training sample sets that can be used by the category mapping relationship. For example, assuming that a category mapping relationship is determined based on the annotation categories involved in training sample sets A and B, then training sample sets A and B are both training sample sets related to the category mapping relationship.

[0120] Because different features should maintain semantic consistency even within different semantic spaces, the server can use the difference between the converted domain-aware query and the unconverted domain-aware query as a reference loss value for model training. Specifically, the server performs semantic space conversion on the domain-aware query corresponding to the training sample set. After obtaining the converted domain-aware query, the server can calculate the difference between the converted and unconverted domain-aware queries as a reference loss value for model training.

[0121] like Figure 3 As shown, the server can determine a reference loss value based on the domain-aware queries of multiple training sample sets using a domain-aware cross-attention module. Specifically, the server can perform semantic space conversion on the domain-aware queries corresponding to each training sample set based on the determined category mapping relationship. Then, based on the domain-aware queries before and after the conversion, the server constructs a reference loss value corresponding to each training sample set.

[0122] As an example, the server can determine the above-mentioned reference loss value in the following manner: combine multiple training sample sets in pairs to obtain at least one training sample set combination; for each training sample set combination, determine one of the training sample sets as the source training sample set, and determine the other training sample set as the target training sample set; according to the semantic features of the annotation categories involved in the source training sample set and the target training sample set, determine the category mapping relationship between the respective annotation systems of the source training sample set and the target training sample set; based on the category mapping relationship, perform semantic space conversion processing on the source domain-aware query corresponding to the source training sample set to obtain a conversion domain-aware query; and then, according to the source domain-aware query and the conversion domain-aware query, determine the reference loss value corresponding to the training sample set combination.

[0123] Specifically, the server may combine multiple training sample sets in pairs to obtain at least one training sample set combination, where each training sample set combination includes two training sample sets.

[0124] For a training sample set combination including training sample set s and training sample set t, the server can use training sample set s as the source training sample set and training sample set t as the target training sample set; accordingly, the domain-aware query corresponding to the source training sample set s is used as the source domain-aware query The domain-aware query corresponding to the target training sample set t is used as the target domain-aware query For the above-mentioned source training sample set s and target training sample set t, the server can determine the semantic features of each annotation category involved in the source training sample set s, and the semantic features of each annotation category involved in the target training sample set t, and determine the category mapping relationship between the respective annotation systems of the source training sample set s and the target training sample set t based on the semantic features of the annotation categories involved in the source training sample set s and the target training sample set t; illustratively, the various annotation categories involved in the source training sample set s can be mapped to a high-dimensional feature space through a language model, and the various annotation categories involved in the target training sample set t can also be mapped to a high-dimensional feature space, and then the inner product of the mapped features can be used as the adjacency matrix representing the category mapping relationship. Among them, N t is the number of labeled categories involved in the target training sample set t, N s is the number of labeled categories involved in the source training sample set s.

[0125] Furthermore, based on the category mapping relationship determined in the above manner, the source domain-aware query can be converted to the semantic space corresponding to the target domain-aware query to obtain a converted domain-aware query; the converted domain-aware query and the source domain-aware query should have semantic consistency, and therefore, the reference loss value corresponding to the training sample set combination can be determined based on the difference between the converted domain-aware query and the source domain-aware query. It should be understood that in actual applications, the source domain-aware query and the target domain-aware query can also be swapped, that is, the target domain-aware query is converted to the semantic space corresponding to the source domain-aware query to obtain a converted domain-aware query, and another reference loss value corresponding to the training sample set combination is determined based on the difference between the converted domain-aware query and the target domain-aware query.

[0126] As an example, the server can specifically convert the source domain-aware query into the semantic space corresponding to the target domain-aware query in the following manner to obtain a converted domain-aware query: use the target domain-aware query corresponding to the target training sample set as the query element in the attention mechanism, and use the source domain-aware query as the key element and value element in the attention mechanism; then, based on the attention mechanism, according to the above-mentioned query elements, key elements and value elements, and category mapping relationships, convert the source domain-aware query into the semantic space corresponding to the target domain-aware query to obtain a converted domain-aware query.

[0127] See also Figure 6 , Figure 6 This is a schematic diagram of the working principle of the domain-aware cross-domain attention module provided in the embodiment of this application. Figure 6 As shown, in the domain-aware cross-domain attention module, the target domain-aware query As the query element in the multi-head cross attention mechanism, the source instance domain-aware query As the key element and value element in the multi-head cross attention mechanism, the category mapping relationship B determined based on the semantic features of the labeled categories involved in the source training sample set and the target training sample set can be global As a mask, based on and The generated attention map completes the semantic space transformation through the cross-attention mechanism.

[0128] Specifically, the output feature of the j-th head in the multi-head cross attention mechanism can be expressed as formula (12):

[0129]

[0130] Among them, W q,j 、W k,j and W v,j is the model parameter associated with the jth head in the multi-head cross-attention mechanism, and d is the dimension of the feature vector in the domain-aware query.

[0131] Then, the output features of each head in the multi-head cross attention mechanism are spliced together by formula (13) to obtain the output result of the multi-head cross attention mechanism:

[0132]

[0133] Then, the output of the multi-head cross attention mechanism is combined with the target domain perception query Add them together to get Q′ N+1 , and Q′ is obtained through layer regularization and feedforward network N+1 Processing is performed, and finally the processing results are compared with Q′ N+1 Add together to get the conversion domain aware query The processing can be expressed as follows:

[0134]

[0135]

[0136] Finally, we can use the following formula (16) to query based on the source domain perception Its corresponding conversion domain aware query Determine the reference loss value L tcr :

[0137]

[0138] In this way, by determining the reference loss value in the above manner, it is possible to ensure accurate semantic space conversion processing of the domain-aware queries corresponding to the training sample set, thereby ensuring the accuracy of the reference loss value determined based on the domain-aware queries before and after the conversion. Training the image segmentation model in combination with this reference loss value is conducive to ensuring the reliability of model training.

[0139] In one possible implementation, in addition to performing semantic space conversion on instance domain-aware queries or domain-aware queries corresponding to the training sample set, and constructing a loss value for the training model based on the pre- and post-conversion instance domain-aware queries or pre- and post-conversion domain-aware queries, the server can also construct a loss value for the training model based on the difference between the predicted and labeled categories corresponding to the training images. That is, for each training image, the server can determine the predicted category corresponding to the training image based on the decoded features of the training image; further, based on the labeled and predicted categories corresponding to the training image, the server can determine a base loss value for the training image.

[0140] Specifically, for each training image, the server can use the decoded features generated when the classifier processes the training image based on the image segmentation model to determine the predicted category corresponding to the training image. This predicted category belongs to the annotation system of the training sample set to which the training image belongs. Furthermore, based on the difference between the predicted category and the annotated category corresponding to the training image, the server can determine the base loss value corresponding to the training image. This base loss value can be combined with the target loss value and reference loss value mentioned above to train the image segmentation model.

[0141] In this way, by determining the basic loss value corresponding to each training image in the above manner, and using the basic loss value corresponding to each training image in combination with the target loss value and the reference loss value to train the image segmentation model, the performance of the trained image segmentation model can be more reliably guaranteed.

[0142] Step 205: Based on the target loss value, train the image segmentation model.

[0143] The server performs semantic space conversion on the instance-domain-aware query corresponding to the training image, and after determining the target loss value based on the instance-domain-aware query before and after the conversion, the image segmentation model can be further trained based on the target loss value. Specifically, the server can use the difference between the instance-domain-aware query before and after the conversion represented by the target loss value as the training target to train the image segmentation model; taking the target loss value as the loss value calculated by formula (11) above as an example, when training the image segmentation model, minimizing the target loss value can be used as the training target. Train the image segmentation model.

[0144] In the case where the server also performs semantic space conversion processing on the domain-aware queries corresponding to the training sample set and determines a reference loss value based on the domain-aware queries before and after the conversion, the server can train the image segmentation model based on the reference loss value and the target loss value. Specifically, the server can use the difference between the instance domain-aware queries before and after the conversion represented by the target loss value and the difference between the domain-aware queries before and after the conversion represented by the reference loss value as training objectives to train the image segmentation model; taking the target loss value as the loss value calculated by formula (11) above and the reference loss value as the loss value calculated by formula (16) above as an example, the server can use the minimization of the target loss value and the minimization of the reference loss value as training objectives to train the image segmentation model.

[0145] When the server also determines the corresponding predicted category for each training image and determines a basic loss value based on the labeled category and predicted category corresponding to each training image, the server can train the image segmentation model based on the basic loss value, the target loss value, and the reference loss value. Specifically, the server can use minimizing the difference between the instance domain-aware queries before and after the conversion represented by the target loss value, minimizing the difference between the domain-aware queries before and after the conversion represented by the reference loss value, and minimizing the difference between the predicted category and the labeled category represented by the basic loss value as training objectives to train the image segmentation model.

[0146] In actual applications, the server can iteratively perform multiple rounds of steps 201 to 205 based on different training images in multiple training sample sets to achieve multiple rounds of iterative training for the image segmentation model until the trained image segmentation model meets the training end condition. The training end condition here can be, for example, the number of iterative training times for the image segmentation model reaches a preset number, or the model performance of the trained image segmentation model reaches a preset performance requirement (such as the accuracy of the image segmentation model reaches a preset accuracy), or the model performance of the trained image segmentation model no longer has a significant improvement (such as the difference between the accuracy of the image segmentation model obtained from multiple rounds of training is less than a preset difference threshold), etc. This application does not impose any restrictions on the training end condition of the image segmentation model.

[0147] It should be noted that Figure 3 The intra-domain cross-attention module, domain-aware cross-domain cross-attention module, and instance-aware cross-domain cross-attention module used when executing the above steps 203 and 204 are only used in the training phase of the image segmentation model. In the testing phase and application phase of the image segmentation model, the above modules will be removed.

[0148] Optionally, the image segmentation model trained by the embodiment of the present application can also be used to perform image segmentation tasks adapted to a specific annotation system, where the specific annotation system includes any one or more annotation systems of the multiple training sample sets used when training the image segmentation model. Specifically, when performing an image segmentation task on a target image, the target image can be encoded by the encoder in the trained image segmentation model to obtain the encoding features of the target image; the encoding features of the target image can be decoded by the decoder in the image segmentation model to obtain the target decoding features of the target image; then, the target decoding features are respectively subjected to category mapping by multiple receptors to obtain multiple category mapping results; the multiple category mapping results here correspond to the annotation systems of multiple training sample sets respectively; when it is necessary to determine the target category prediction result of the target image under the target annotation system, the target category prediction result can be determined based on these multiple category mapping results and the semantic conversion relationship between the target annotation system and other annotation systems.

[0149] For example, assume that the image segmentation model is obtained by jointly training the training sample set s and the training sample set t, and the number of labeled categories involved in the training sample set s is N S , the number of labeled categories involved in the training sample set t is N T After completing the training of the image segmentation model using the training sample set s and the training sample set t, the image segmentation model can be used to process the target image; when the image segmentation model specifically processes the target image, it will first encode the target image through the encoder therein to obtain the encoding features of the target image, and then decode the encoding features of the target image through the decoder therein to obtain the target decoding features of the target image. The target decoding features are the decoding features with the largest feature scale output by the last decoder in the image segmentation model. The target decoding features can be specifically expressed as

[0150] Then, the server can use multiple sensors including multiple layers to perform category mapping on the target decoding features, and map the dimensions of the target decoding features to the annotation systems corresponding to different training sample sets, that is, Mapped to and Among them G S is the category mapping result corresponding to the annotation system of the training sample set S, G T is the category mapping result corresponding to the annotation system of the training sample set T. The above multiple sensors can be integrated into Figure 3 The feature integration module shown.

[0151] Furthermore, the server may determine the target category prediction result of the target image under the target labeling system based on the category mapping results of the target image under multiple labeling systems and the semantic conversion relationship between the target labeling system and other labeling systems.

[0152] When specifically determining the target category prediction result, the server can determine the conversion category mapping result corresponding to each category mapping result except the target category mapping result corresponding to the target annotation system based on the category mapping result and the semantic conversion relationship between the annotation system corresponding to the category mapping result and the target annotation system; the semantic conversion relationship here is determined based on the domain-aware queries corresponding to the annotation system corresponding to the category mapping result and the target annotation system; and further, the above-mentioned target category prediction result is determined based on the target category mapping result and the conversion category mapping results corresponding to each category mapping result except the target category mapping result.

[0153] For example, for training sample set s and training sample set t, the server can determine the semantic conversion relationship from the annotation system of training sample set s to the annotation system of training sample set t, and the semantic conversion relationship from the annotation system of training sample set t to the annotation system of training sample set s, respectively, through the following formulas (17) and (18). The determined semantic conversion relationship is specifically expressed as the conversion matrix V S→T and V T→S :

[0154]

[0155]

[0156] in, and are the domain-aware queries corresponding to the training sample set s and the training sample set t, respectively. and are matrix transformation parameters.

[0157] When the annotation system corresponding to the training sample set S is the target annotation system, the server can determine the target category prediction result Segment of the target image under the annotation system corresponding to the training sample set s by the following formula (19): S :

[0158] Segment S =FFN S (Concat(G S ,G T ·V T→S )) (19)

[0159] Among them, G Sis the target category mapping result, G T ·V T→S The category mapping result G T The corresponding conversion category mapping result.

[0160] When the annotation system corresponding to the training sample set t is the target annotation system, the server can determine the target category prediction result Segment of the target image under the annotation system corresponding to the training sample set t by the following formula (20): T :

[0161] Segment T =FFN T (Concat(G T ,G S ·V S→T )) (20)

[0162] Among them, G T is the target category mapping result, G S ·V S→T The category mapping result G S The corresponding conversion category mapping result.

[0163] In this way, through the above method, the image segmentation model trained by the embodiment of the present application can be adapted to segmentation tasks of different precisions, that is, the image segmentation model can accurately perform segmentation tasks in the labeling systems corresponding to multiple training sample sets, thereby meeting the scenario requirements of an image segmentation model taking into account multiple different precision requirements.

[0164] The above-mentioned model training method performs semantic mapping processing on the decoding features generated when the image segmentation model to be trained processes the training image, and obtains the instance category features that reflect the mapping relationship between the decoding features and the labeled categories involved in the training sample set to which the training image belongs, and then mines the semantic conversion relationship between the instance category features corresponding to different training images, and accordingly performs semantic space conversion processing on the instance domain perception query corresponding to the semantic space of the instance category feature. Taking into account that the instance domain perception query before and after the semantic space conversion processing should have semantic consistency, the embodiment of the present application accordingly constructs a target loss value for training the image segmentation model according to the difference between the instance domain perception queries before and after the conversion, and trains the image segmentation model based on the target loss value. The image segmentation model trained in the above manner can achieve good segmentation effects on each training sample set used.

[0165] In order to facilitate further understanding of the model training method provided in the embodiment of the present application, the following uses the example of using two training sample sets to jointly train a human segmentation model to provide an overall exemplary introduction to the model training method provided in the embodiment of the present application.

[0166] The model training method provided in the embodiment of the present application is suitable for training a human body segmentation model based on an encoder-decoder architecture built on a Transformer module. The training framework of the human body segmentation model is as follows: Figure 3 When training this human segmentation model, the encoder is retained, and domain-aware query is introduced in the decoding stage. By semantically structuring the annotated categories, semantic consistency is mined from both the domain and instance dimensions, thereby implicitly embedding the semantic information involved in different training sample sets into a universal human segmentation model.

[0167] Figure 3 The lower half of the figure is a human segmentation model based on the U-Net structure, in which the encoder is used to extract features and the decoder is used to generate segmentation results step by step. When training the human segmentation model, the decoded features of each feature scale output by the decoder are mapped to the annotation system involved in the corresponding training sample set to generate a semantic mapping of the decoded features, that is, the instance category features; and the instance category features are interacted with the domain-aware query through the intra-domain cross-attention module to generate an instance domain-aware query corresponding to each feature scale. The instance domain-aware query is used as the input of the intra-domain cross-attention module of the next feature scale and interacts with the instance category features corresponding to the next feature scale. In addition, the domain-aware query corresponding to the training sample set and the instance domain-aware query corresponding to each feature scale can be converted into each other in different semantic spaces through the domain-aware cross-domain cross-attention module and the instance-aware cross-domain cross-attention module respectively. Then, the loss value used for model training is determined based on semantic consistency, and the model training is supervised together with the segmentation loss function (that is, the loss function constructed based on the difference between the annotation result and the prediction result).

[0168] Below Figure 3 Each part involved in the model training architecture shown is introduced in detail.

[0169] Encoder-Decoder Architecture:

[0170] When two training sample sets are used to jointly train the human segmentation model, the input of each iteration is a training sample set from two different annotation systems {D s ,D t} training images In the encoding stage, the feature map can be processed by halving the resolution and doubling the number of channels through the Transformer module and downsampling operation. After N downsampling, N+1 sets of encoding features with different feature scales will be obtained. The encoding feature corresponding to feature scale i can be expressed as In the decoding stage, we can start from the encoding features with the smallest resolution, use the Transformer module and upsampling operation to increase the resolution of the feature map and halve the number of channels; after the decoder outputs the decoded features, we can add the decoded features to the encoding features of the same feature scale to obtain the input features of the next decoder.

[0171] Generation of instance category features:

[0172] The purpose of this step is to generate a semantic mapping of the decoded features in the semantic space. The corresponding decoding feature with feature scale λ can be expressed as When performing semantic mapping based on the decoding features, we can first use the mapping matrix Calculate the weight of the decoded feature for each category (each labeled category involved in the training sample set to which the training image belongs), where N is the total number of labeled categories involved in the training sample set; then, calculate the reference instance category feature corresponding to the feature scale λ based on the weight; and then, use the transformation matrix Convert the feature dimensions corresponding to each category in the reference instance category feature to d, and obtain the instance category feature corresponding to the feature scale λ of the training image I The above process of generating instance category features can be specifically shown as follows:

[0173]

[0174] Determination of domain-aware queries:

[0175] In the embodiment of the present application, a corresponding domain-aware query is defined for each training sample set used to train the human segmentation model. Where N is the number of labeled categories involved in the training sample set, and d is the dimension of the category feature vector. The domain-aware query Q corresponding to the training sample set and the instance category feature Z corresponding to the minimum feature scale I,N Interact to generate the instance domain-aware query corresponding to the minimum feature scale, which will be used as input to the instance category feature Z corresponding to the next feature scale I,N-1 Through the interaction between the domain-aware query and the instance category features corresponding to each feature scale, we can obtain the domain-aware query Q corresponding to the training sample set and the instance domain-aware query {Q1, Q2, ..., Q N}, in order to unify the representation, the domain-aware query corresponding to the training sample set can be regarded as the instance domain-aware query corresponding to the N+1th feature scale, denoted as Q N+1 , then N+1 instance domain-aware queries can be expressed as

[0176] How the intra-domain criss-cross attention module works:

[0177] For the feature scale λ, the intra-domain cross-attention module can take the instance domain-aware query Q corresponding to the previous feature scale λ+1 as λ+1 As the query element in the multi-head cross attention mechanism, the instance category feature Z corresponding to the feature scale λ is I,λ As key elements and value elements in the multi-head cross attention mechanism; the category characteristics of the feature scale λ are enhanced by the cross attention mechanism of the query element and the key element. In addition, the semantic relationship between categories is used to construct the adjacency matrix Representing the semantic association between the labeled categories involved in the training image, in the adjacency matrix A Intra In, a i,j =1 means that the human body parts corresponding to the i-th and j-th annotation categories are adjacent, a i,j = 0 means that the human body parts corresponding to the i-th label category and the j-th label category are not adjacent. Intra Acts as a mask based on Q λ+1 and Z I,λ The generated attention map embeds the semantic prior knowledge between the labeled categories; then, the features fused with instance information are further processed by layer regularization and feedforward neural network, and the processing results are consistent with the reference instance domain perception query Q λ+1 Add together to generate the instance domain-aware query Q corresponding to the feature scale λ λ .

[0178] The working of the cross-attention module in this domain can be found in Figure 4 , which can be specifically expressed by the following formula:

[0179]

[0180] IntraMHCA(Q λ+1 ,Z I,λ ,A intra )=concat(IntraCA1,…,IntraCA M )

[0181] Q′ λ =IntraMHCA(Q λ+1 ,Z I,λ ,A intra )+Q λ+1

[0182] Q λ =FFN(Norm(Q′ λ ))+Q′ λ

[0183] Among them, IntraCA(Q λ+1 ,Z I,λ ,A intra ) is the output of the j-th head in the multi-head cross attention mechanism, IntraMHCA(Q λ+1 ,Z I,λ ,A intra ) is the overall output result of the multi-head cross attention mechanism.

[0184] The working principle of the cross-domain attention module (including the domain-aware cross-domain attention module and the instance-aware cross-domain attention module):

[0185] In addition to the semantic correlation between categories in the same semantic space, the embodiment of the present application further cross-maps the semantic features of each training sample set to the semantic space of other training sample sets, and then uses the semantic consistency of the same semantic feature in different semantic spaces as a constraint condition to assist model training. and are the domain-aware query sets of training sample sets s and t, respectively, N s is the number of labeled categories involved in the training sample set s, N t is the number of labeled categories involved in the training sample set t. In each domain-aware query set, the first N are instance-domain-aware queries, which can be interacted using the instance-aware cross-domain cross-attention module. and They are the domain-aware queries corresponding to the training sample set s and the training sample set t, respectively, and can interact using the domain-aware cross-domain attention module.

[0186] Taking the mapping of the domain-aware query corresponding to the training sample set s to the semantic space of the training sample set t as an example, the domain-aware query corresponding to the training sample set t is mapped to the semantic space of the training sample set t. As the query element in the multi-head cross attention mechanism, the domain-aware query corresponding to the training sample set s is As the key element and value element in the multi-head cross attention mechanism. Through the language model, the various labeled categories involved in the training sample set s and the training sample set t are mapped to the high-dimensional feature space, and the inner product of the mapped features is used as the adjacency matrix representing the category mapping relationship. The B global Acts as a mask based on and The generated attention map.

[0187] The working method of the domain-aware cross-domain attention module can be found in Figure 6 , which can be specifically expressed by the following formula:

[0188]

[0189]

[0190]

[0191]

[0192] For the same part of the human body, the mapped domain-aware query in the semantic space corresponding to the training sample set t and the original domain-aware query in the semantic space corresponding to the training sample set s are semantically consistent. This consistency can be enforced by maximizing the cosine distance between the two domain-aware queries. That is, the reference loss value used in training the model can be determined by the following formula:

[0193]

[0194] For instance-domain-aware queries corresponding to each feature scale, the present embodiment designs an instance-aware cross-domain attention module. s and I t The instance domain-aware queries corresponding to the feature scale λ from the training sample set s and the training sample set t are respectively expressed as and In the instance-aware cross-domain attention module, we can As the query element in the multi-head cross attention mechanism, As the key element and value element in the multi-head cross attention mechanism. Use the instance category features corresponding to the feature scale λ to calculate the adjacency matrix Represents the semantic transformation relationship based on instance category features and uses it as a mask based on and Determined domain attention map.

[0195]

[0196] After completing the semantic space conversion processing through the multi-head cross-attention mechanism, the semantic consistency of the same category can be used as the loss value to assist model training.

[0197] The working method of this instance-aware cross-domain attention module can be found in Figure 5 , which can be specifically expressed by the following formula:

[0198]

[0199]

[0200]

[0201]

[0202] Furthermore, the target loss value used when training the model can be determined by the following formula:

[0203]

[0204] How the feature integration module works:

[0205] After decoding is completed by the trained human body segmentation model, the embodiment of the present application can also perform segmentation tasks of different precisions according to the decoding features generated by the human body segmentation model processing the input image through the feature integration module. The segmentation tasks of different precisions here can specifically include tasks S and tasks T, wherein task S is used to determine the input image in N according to the decoding features. S The segmentation results under N categories, task T is used to determine the input image according to the decoding features. T Segmentation results under categories.

[0206] The input image is processed by the encoder and decoder in the human segmentation model to obtain the decoding features Use multi-layer receptors to map the dimension of F to the number of labeled categories involved in the training sample set s and the training sample set t, and get and In addition, based on the domain-aware query corresponding to the training sample set s and the training sample set t and The transformation matrix V between the segmentation feature maps corresponding to different training sample sets can be calculated by the following formula S→T and V T→S :

[0207]

[0208]

[0209] Furthermore, G can be expressed as S and G T Perform integration to obtain segmentation results with different accuracies:

[0210] Segment S =FFN S (Concat(G S ,G T ·V T→S ))

[0211] Segment T =FFN T (Concat(G T ,G S ·V T→S ))

[0212] Among them, Segment S is the segmentation result under the annotation system of the training sample set s, Segment T is the segmentation result under the annotation system of the training sample set t.

[0213] Among the above modules, the intra-domain cross-attention module, the domain-aware cross-domain cross-attention module, and the instance-aware cross-domain cross-attention module are only used in the model training phase and will be removed in the model testing and application phases.

[0214] To verify the effectiveness and reliability of the method provided in the embodiments of the present application, the inventors of this application used SwinUNet as a benchmark model and verified the model performance on three datasets: PASCAL Person-Part, ATR, and CIHP. The mean intersection-over-union (mIoU) was used as the measurement indicator, and the verification results were shown in Table 1 below:

[0215] Table 1

[0216]

[0217]

[0218] Among them, Swin UNet-1 is an image segmentation model trained using a single training sample set, Swin UNet-2 is an image segmentation model trained using three training sample sets after directly adding three segmentation heads to the model, Graphonomy is an existing solution that optimizes the joint training of multiple training sample sets by mining semantic information, and SST is an image segmentation model trained using the method provided in the embodiments of this application. Through comparison, it can be found that the image segmentation model trained using the method provided in the embodiments of this application achieved the highest mIoU on all datasets.

[0219] In response to the model training method described above, this application also provides a corresponding model training device to enable the above model training method to be applied and implemented in practice.

[0220] See also Figure 7 , Figure 7 It is the same as above Figure 2 The structural diagram of a model training device 700 corresponding to the model training method shown in FIG. Figure 7 As shown, the model training device 700 includes:

[0221] An acquisition module 701 is configured to acquire a plurality of training images from a plurality of training sample sets, and acquire a domain-aware query corresponding to each of the plurality of training sample sets; the domain-aware query corresponds to a semantic space of a labeling system of the corresponding training sample set;

[0222] The image processing module 702 is configured to encode the plurality of training images using an encoder in the image segmentation model to obtain encoding features of the plurality of training images; and decode the encoding features of the plurality of training images using a decoder in the image segmentation model to obtain decoding features of the plurality of training images.

[0223] The feature processing module 703 is configured to perform semantic mapping on each of the training images based on the decoded features of the training image to obtain instance category features corresponding to the training image; determine an instance domain-aware query corresponding to the training image based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category features; the instance domain-aware query corresponds to the semantic space of the instance category features;

[0224] The loss value determination module 704 is configured to determine a semantic conversion relationship between instance category features corresponding to each of the plurality of training images; based on the semantic conversion relationship, perform semantic space conversion processing on the instance domain-aware query corresponding to the training image; and determine a target loss value based on the instance domain-aware query before and after the conversion;

[0225] The model training module 705 is used to train the image segmentation model based on the target loss value.

[0226] Optionally, the decoding features of each training image include decoding features of N feature scales, where N is an integer greater than 1; the feature processing module 703 is specifically configured to:

[0227] For each of the feature scales, semantic mapping is performed according to the decoded features of the feature scale to obtain instance category features corresponding to the feature scale;

[0228] Determining instance domain-aware queries corresponding to each of the N feature scales based on a domain-aware query corresponding to a training sample set to which the training image belongs and instance category features corresponding to each of the N feature scales; there being a corresponding relationship between instance category features corresponding to the same feature scale and instance domain-aware queries;

[0229] The loss value determination module 704 is specifically configured to:

[0230] For the target feature scale, a semantic conversion relationship corresponding to the target feature scale is determined based on the instance category features of each of the multiple training images corresponding to the target feature scale; based on the semantic conversion relationship corresponding to the target feature scale, a semantic space conversion process is performed on the instance domain-aware query corresponding to the target feature scale of the training image, and a target loss value corresponding to the target feature scale is determined based on the instance domain-aware query before and after the conversion.

[0231] Optionally, the loss value determination module 704 is specifically configured to:

[0232] constructing at least one instance-domain-aware query combination based on the instance-domain-aware queries corresponding to the target feature scale, respectively, of the plurality of training images; each instance-domain-aware query combination includes two instance-domain-aware queries;

[0233] For each instance domain-aware query combination, determine one of the instance domain-aware queries as the source instance domain-aware query, and determine the other instance domain-aware query as the target instance domain-aware query; determine the semantic conversion relationship based on the instance category features corresponding to the source instance domain-aware query and the target instance domain-aware query; based on the semantic conversion relationship, convert the source instance domain-aware query to the semantic space corresponding to the target instance domain-aware query to obtain a converted instance domain-aware query; determine the target loss value corresponding to the instance domain-aware query combination based on the source instance domain-aware query and the converted instance domain-aware query.

[0234] Optionally, the loss value determination module 704 is specifically configured to:

[0235] Using the target instance domain-aware query as a query element in an attention mechanism, and using the source instance domain-aware query as a key element and a value element in the attention mechanism;

[0236] Based on the attention mechanism, the source instance domain-aware query is converted to the semantic space corresponding to the target instance domain-aware query according to the query element, the key element, the value element, and the semantic conversion relationship to obtain the converted instance domain-aware query.

[0237] Optionally, the feature processing module 703 is specifically configured to:

[0238] For each of the feature scales, determining an instance domain-aware query corresponding to the feature scale according to a reference instance domain-aware query corresponding to the feature scale and an instance category feature corresponding to the feature scale;

[0239] Among them, the reference instance domain-aware query corresponding to the minimum feature scale among the N feature scales is the domain-aware query corresponding to the training sample set; the reference instance domain-aware query corresponding to each feature scale among the N feature scales except the minimum feature scale is the instance domain-aware query corresponding to the feature scale among the N feature scales that is smaller than the feature scale and adjacent to the feature scale.

[0240] Optionally, the feature processing module 703 is specifically configured to:

[0241] Determining the semantic association relationship between each labeled category involved in the training image;

[0242] Using the reference instance domain-aware query as a query element in an attention mechanism, and using the instance category feature as a key element and a value element in the attention mechanism;

[0243] Based on the attention mechanism, the instance domain-aware query is determined according to the query element, the key element, the value element, and the semantic association relationship.

[0244] Optionally, the loss value determination module 704 is further configured to:

[0245] Determining a category mapping relationship between the annotation systems of the multiple training sample sets; performing semantic space conversion processing on the domain-aware queries corresponding to the training sample sets based on the category mapping relationship, and determining a reference loss value based on the domain-aware queries before and after the conversion;

[0246] The model training module 705 is specifically used to:

[0247] The image segmentation model is trained based on the target loss value and the reference loss value.

[0248] Optionally, the loss value determination module 704 is specifically configured to:

[0249] Combining the plurality of training sample sets in pairs to obtain at least one training sample set combination;

[0250] For each combination of the training sample sets, one of the training sample sets is determined to be the source training sample set, and the other training sample set is determined to be the target training sample set; based on the semantic features of the annotation categories involved in the source training sample set and the target training sample set, a category mapping relationship between the respective annotation systems of the source training sample set and the target training sample set is determined; based on the category mapping relationship, a semantic space conversion process is performed on the source domain-aware query corresponding to the source training sample set to obtain a converted domain-aware query; based on the source domain-aware query and the converted domain-aware query, a reference loss value corresponding to the training sample set combination is determined.

[0251] Optionally, the loss value determination module 704 is specifically configured to:

[0252] Using the target domain-aware query corresponding to the target training sample set as the query element in the attention mechanism, and using the source domain-aware query as the key element and value element in the attention mechanism;

[0253] Based on the attention mechanism, the source domain-aware query is converted into a semantic space corresponding to the target domain-aware query according to the query element, the key element, the value element, and the category mapping relationship to obtain the converted domain-aware query.

[0254] Optionally, the loss value determination module 704 is further configured to:

[0255] For each of the training images, determining a predicted category corresponding to the training image based on the decoding features of the training image; determining a basic loss value corresponding to the training image based on the labeled category corresponding to the training image and the predicted category;

[0256] The model training module 705 is specifically used to:

[0257] The image segmentation model is trained based on the target loss value and the base loss values corresponding to each of the plurality of training images.

[0258] Optionally, the device further includes a model application module, wherein the model application module is configured to:

[0259] The target image is encoded by the encoder in the image segmentation model to obtain the encoding features of the target image; the encoding features of the target image are decoded by the decoder in the image segmentation model to obtain the target decoding features of the target image;

[0260] Performing category mapping processing on the target decoding features through multiple sensors to obtain multiple category mapping results; the multiple category mapping results respectively correspond to the labeling systems of the multiple training sample sets;

[0261] When it is necessary to determine the target category prediction result of the target image under the target labeling system, the target category prediction result is determined according to the multiple category mapping results and the semantic conversion relationship between the target labeling system and other labeling systems.

[0262] Optionally, the model application module is specifically used to:

[0263] For each category mapping result other than the target category mapping result corresponding to the target annotation system, determining a converted category mapping result corresponding to the category mapping result based on the category mapping result and a semantic conversion relationship between the annotation system corresponding to the category mapping result and the target annotation system; the semantic conversion relationship is determined based on domain-aware queries corresponding to the annotation system corresponding to the category mapping result and the target annotation system, respectively.

[0264] The target category prediction result is determined according to the target category mapping result and the conversion category mapping results corresponding to each category mapping result other than the target category mapping result.

[0265] The above-mentioned model training device performs semantic mapping processing on the decoding features generated when the image segmentation model to be trained processes the training image, and obtains the instance category features that reflect the mapping relationship between the decoding features and the labeled categories involved in the training sample set to which the training image belongs, and then mines the semantic conversion relationship between the instance category features corresponding to different training images, and accordingly performs semantic space conversion processing on the instance domain perception query corresponding to the semantic space of the instance category feature. Taking into account that the instance domain perception query before and after the semantic space conversion processing should have semantic consistency, the embodiment of the present application accordingly constructs a target loss value for training the image segmentation model according to the difference between the instance domain perception queries before and after the conversion, and trains the image segmentation model based on the target loss value. The image segmentation model trained by the above-mentioned device can achieve good segmentation effect on each training sample set used.

[0266] An embodiment of the present application also provides a computer device for training a model, which may specifically be a terminal device or a server. The terminal device and server provided in the embodiment of the present application will be introduced below from the perspective of hardware instantiation.

[0267] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of the terminal device provided in the embodiment of the present application. Figure 8 For ease of explanation, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, tablet computer, personal digital assistant (PDA), point of sales (POS), car computer, etc. For example, the terminal is a computer:

[0268] Figure 8 FIG2 is a block diagram showing a partial structure of a computer related to a terminal provided in an embodiment of the present application. Figure 8 The computer includes: a radio frequency (RF) circuit 810, a memory 820, an input unit 830 (including a touch panel 831 and other input devices 832), a display unit 840 (including a display panel 841), a sensor 850, an audio circuit 860 (which can be connected to a speaker 861 and a microphone 862), a wireless fidelity (WiFi) module 870, a processor 880, and a power supply 890. Those skilled in the art will understand that Figure 8 The computer structure shown in the figure does not constitute a limitation of the computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0269] The memory 820 can be used to store software programs and modules. The processor 880 executes the various computer functions and data processing by running the software programs and modules stored in the memory 820. The memory 820 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the computer (such as audio data, a phone book, etc.). In addition, the memory 820 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0270] The processor 880 is the computer's control center, connecting all computer components using various interfaces and circuits. It executes the computer's various functions and processes data by running or executing software programs and / or modules stored in the memory 820 and accessing data stored in the memory 820. Optionally, the processor 880 may include one or more processing units. Preferably, the processor 880 integrates an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 880.

[0271] In an embodiment of the present application, the processor 880 included in the terminal is also used to execute the steps of any implementation method of the model training method provided in the embodiment of the present application.

[0272] See also Figure 9 , Figure 9A structural diagram of a server 900 provided for an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 922 (for example, one or more processors) and a memory 932, and one or more storage media 930 (for example, one or more massive storage devices) for storing application programs 942 or data 944. Among them, the memory 932 and the storage medium 930 may be temporary storage or permanent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 922 may be configured to communicate with the storage medium 930 to execute a series of instruction operations in the storage medium 930 on the server 900.

[0273] The server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 958, and / or one or more operating systems, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0274] The steps performed by the server in the above embodiment can be based on the Figure 9 The server structure shown.

[0275] Among them, CPU 922 is used to execute the steps of any implementation method of the model training method provided in the embodiments of the present application.

[0276] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program, which is used to execute any implementation of a model training method described in the aforementioned embodiments.

[0277] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any one of the implementations of the model training method described in the aforementioned embodiments.

[0278] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0279] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0280] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0281] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0282] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store computer programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0283] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0284] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a plurality of training images from a plurality of training sample sets, and acquire domain-aware queries corresponding to the plurality of training sample sets; the domain-aware queries correspond to the semantic space of the annotation system of the corresponding training sample sets; The plurality of training images are respectively encoded by an encoder in the image segmentation model to obtain encoding features of the plurality of training images; the encoding features of the plurality of training images are respectively decoded by a decoder in the image segmentation model to obtain decoding features of the plurality of training images; For each of the training images, semantic mapping is performed based on the decoded features of the training image to obtain instance category features corresponding to the training image; based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category features, an instance domain-aware query corresponding to the training image is determined; the instance domain-aware query corresponds to the semantic space of the instance category features; Determining a semantic conversion relationship between instance category features corresponding to each of the plurality of training images; performing semantic space conversion processing on the instance domain-aware query corresponding to the training image based on the semantic conversion relationship, and determining a target loss value based on the instance domain-aware query before and after the conversion; The image segmentation model is trained based on the target loss value.

2. The method according to claim 1, characterized in that The decoding features of each training image include decoding features of N feature scales, where N is an integer greater than 1; performing semantic mapping based on the decoding features of the training image to obtain instance category features corresponding to the training image; Determining, according to the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category feature, an instance domain-aware query corresponding to the training image, comprising: For each of the feature scales, semantic mapping is performed according to the decoded features of the feature scale to obtain instance category features corresponding to the feature scale; Determining instance domain-aware queries corresponding to each of the N feature scales based on a domain-aware query corresponding to a training sample set to which the training image belongs and instance category features corresponding to each of the N feature scales; there being a corresponding relationship between instance category features corresponding to the same feature scale and instance domain-aware queries; The determining of the semantic conversion relationship between the instance category features corresponding to each of the plurality of training images; performing semantic space conversion processing on the instance domain-aware query corresponding to the training image based on the semantic conversion relationship, and determining a target loss value based on the instance domain-aware query before and after the conversion, includes: For the target feature scale, a semantic conversion relationship corresponding to the target feature scale is determined based on the instance category features of each of the multiple training images corresponding to the target feature scale; based on the semantic conversion relationship corresponding to the target feature scale, a semantic space conversion process is performed on the instance domain-aware query corresponding to the target feature scale of the training image, and a target loss value corresponding to the target feature scale is determined based on the instance domain-aware query before and after the conversion.

3. The method according to claim 2, characterized in that The method further comprises: determining a semantic conversion relationship corresponding to the target feature scale based on instance category features of each of the plurality of training images corresponding to the target feature scale; performing semantic space conversion processing on the instance domain-aware query corresponding to the target feature scale of the training image based on the semantic conversion relationship corresponding to the target feature scale; and determining a target loss value corresponding to the target feature scale based on the instance domain-aware query before and after the conversion, including: constructing at least one instance-domain-aware query combination based on the instance-domain-aware queries corresponding to the target feature scale, respectively, of the plurality of training images; each instance-domain-aware query combination includes two instance-domain-aware queries; For each instance domain-aware query combination, determine one of the instance domain-aware queries as the source instance domain-aware query, and determine the other instance domain-aware query as the target instance domain-aware query; determine the semantic conversion relationship based on the instance category features corresponding to the source instance domain-aware query and the target instance domain-aware query; based on the semantic conversion relationship, convert the source instance domain-aware query to the semantic space corresponding to the target instance domain-aware query to obtain a converted instance domain-aware query; determine the target loss value corresponding to the instance domain-aware query combination based on the source instance domain-aware query and the converted instance domain-aware query.

4. The method according to claim 3, characterized in that The converting the source instance domain-aware query to a semantic space corresponding to the target instance domain-aware query based on the semantic conversion relationship to obtain a converted instance domain-aware query includes: Using the target instance domain-aware query as a query element in an attention mechanism, and using the source instance domain-aware query as a key element and a value element in the attention mechanism; Based on the attention mechanism, the source instance domain-aware query is converted to the semantic space corresponding to the target instance domain-aware query according to the query element, the key element, the value element, and the semantic conversion relationship to obtain the converted instance domain-aware query.

5. The method according to claim 2, characterized in that The determining, based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category features corresponding to each of the N feature scales, the instance domain-aware query corresponding to each of the N feature scales includes: For each of the feature scales, determining an instance domain-aware query corresponding to the feature scale according to a reference instance domain-aware query corresponding to the feature scale and an instance category feature corresponding to the feature scale; Among them, the reference instance domain-aware query corresponding to the minimum feature scale among the N feature scales is the domain-aware query corresponding to the training sample set; the reference instance domain-aware query corresponding to each feature scale among the N feature scales except the minimum feature scale is the instance domain-aware query corresponding to the feature scale among the N feature scales that is smaller than the feature scale and adjacent to the feature scale.

6. The method according to claim 5, characterized in that The determining, based on the reference instance domain-aware query corresponding to the feature scale and the instance category feature corresponding to the feature scale, the instance domain-aware query corresponding to the feature scale includes: Determining the semantic association relationship between each labeled category involved in the training image; Using the reference instance domain-aware query as a query element in an attention mechanism, and using the instance category feature as a key element and a value element in the attention mechanism; Based on the attention mechanism, the instance domain-aware query is determined according to the query element, the key element, the value element, and the semantic association relationship.

7. The method according to claim 1, characterized in that The method further comprises: Determining a category mapping relationship between the annotation systems of the multiple training sample sets; performing semantic space conversion processing on the domain-aware queries corresponding to the training sample sets based on the category mapping relationship, and determining a reference loss value based on the domain-aware queries before and after the conversion; The step of training the image segmentation model based on the target loss value includes: The image segmentation model is trained based on the target loss value and the reference loss value.

8. The method according to claim 7, characterized in that The step of determining a category mapping relationship between the annotation systems of the plurality of training sample sets; performing semantic space conversion processing on the domain-aware queries corresponding to the training sample sets based on the category mapping relationship, and determining a reference loss value based on the domain-aware queries before and after the conversion, includes: Combining the plurality of training sample sets in pairs to obtain at least one training sample set combination; For each combination of the training sample sets, one of the training sample sets is determined to be the source training sample set, and the other training sample set is determined to be the target training sample set; based on the semantic features of the annotation categories involved in the source training sample set and the target training sample set, a category mapping relationship between the respective annotation systems of the source training sample set and the target training sample set is determined; based on the category mapping relationship, a semantic space conversion process is performed on the source domain-aware query corresponding to the source training sample set to obtain a converted domain-aware query; based on the source domain-aware query and the converted domain-aware query, a reference loss value corresponding to the training sample set combination is determined.

9. The method according to claim 8, characterized in that The step of performing semantic space conversion processing on the source domain-aware query corresponding to the source training sample set based on the category mapping relationship to obtain a converted domain-aware query includes: Using the target domain-aware query corresponding to the target training sample set as the query element in the attention mechanism, and using the source domain-aware query as the key element and value element in the attention mechanism; Based on the attention mechanism, the source domain-aware query is converted into a semantic space corresponding to the target domain-aware query according to the query element, the key element, the value element, and the category mapping relationship to obtain the converted domain-aware query.

10. The method according to claim 1 or 7, characterized in that The method further comprises: For each of the training images, determining a predicted category corresponding to the training image based on the decoding features of the training image; determining a basic loss value corresponding to the training image based on the labeled category corresponding to the training image and the predicted category; The step of training the image segmentation model based on the target loss value includes: The image segmentation model is trained based on the target loss value and the base loss values corresponding to each of the plurality of training images.

11. The method according to claim 1, wherein The method further comprises: The target image is encoded by the encoder in the image segmentation model to obtain the encoding features of the target image; the encoding features of the target image are decoded by the decoder in the image segmentation model to obtain the target decoding features of the target image; Performing category mapping processing on the target decoding features through multiple sensors to obtain multiple category mapping results; the multiple category mapping results respectively correspond to the labeling systems of the multiple training sample sets; When it is necessary to determine the target category prediction result of the target image under the target labeling system, the target category prediction result is determined according to the multiple category mapping results and the semantic conversion relationship between the target labeling system and other labeling systems.

12. The method according to claim 11, characterized in that Determining the target category prediction result according to the multiple category mapping results and the semantic conversion relationship between the target annotation system and other annotation systems includes: For each category mapping result other than the target category mapping result corresponding to the target annotation system, determining a converted category mapping result corresponding to the category mapping result based on the category mapping result and a semantic conversion relationship between the annotation system corresponding to the category mapping result and the target annotation system; the semantic conversion relationship is determined based on domain-aware queries corresponding to the annotation system corresponding to the category mapping result and the target annotation system, respectively. The target category prediction result is determined according to the target category mapping result and the conversion category mapping results corresponding to each category mapping result other than the target category mapping result.

13. A model training device, characterized in that: The device comprises: an acquisition module, configured to acquire a plurality of training images respectively from a plurality of training sample sets, and acquire a domain-aware query corresponding to each of the plurality of training sample sets; the domain-aware query corresponding to the semantic space of the annotation system of the corresponding training sample set; an image processing module, configured to encode the plurality of training images using an encoder in the image segmentation model to obtain encoding features of the plurality of training images; and decode the encoding features of the plurality of training images using a decoder in the image segmentation model to obtain decoding features of the plurality of training images; a feature processing module configured to perform semantic mapping on each of the training images based on the decoded features of the training images to obtain instance category features corresponding to the training images; determine an instance domain-aware query corresponding to the training image based on the domain-aware query corresponding to the training sample set to which the training image belongs and the instance category features; the instance domain-aware query corresponds to the semantic space of the instance category features; a loss value determination module, configured to determine a semantic conversion relationship between instance category features corresponding to each of the plurality of training images; based on the semantic conversion relationship, perform semantic space conversion processing on the instance domain-aware query corresponding to the training image, and determine a target loss value based on the instance domain-aware query before and after the conversion; A model training module is used to train the image segmentation model based on the target loss value.

14. A computer device, characterized in that: The device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the model training method according to any one of claims 1 to 12 according to the computer program.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the model training method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a processor, the model training method according to any one of claims 1 to 12 is implemented.

Images