Image processing method, apparatus, readable storage medium and program product

Abstract

The application relates to an image processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises: performing feature extraction on a target sample image through an image segmentation model to obtain first image features and second image features; determining target embedding features based on the first image features and the second image features; the target embedding features are obtained by fusing first attention features and second attention features, and the first attention features and the second attention features are determined based on the first image features and the second image features; performing attention processing on the target embedding features, reference embedding features of a reference image and mask embedding features of a mask image corresponding to the reference image to obtain aligned embedding features; training the image segmentation model based on the aligned embedding features, and processing a to-be-segmented image based on the trained image segmentation model. The method can effectively improve the fineness of image segmentation processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer and internet technologies, image processing is widely used in civilian and commercial fields. Image segmentation plays an increasingly important role in video surveillance, virtual reality (VR) and other fields, and also faces higher requirements.

[0003] However, current mainstream image segmentation methods often have poor fuzzy segmentation capabilities. For example, in the field of skin measurement, the face may have many spots, acne, wrinkles, etc., making it impossible to achieve fine segmentation. In particular, it is impossible to segment all related lesions, especially wrinkles and fine lines in the field of skin measurement, resulting in poor visual effects of the generated segmented images. Summary of the Invention

[0004] Based on this, this application provides an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product, which can effectively improve the precision of image segmentation processing. It effectively solves the problem that traditional methods cannot perform fine segmentation of wrinkles and fine lines in the skin measurement field, thereby effectively improving the visual effect of image segmentation processing, while also enhancing the user experience and bringing convenience to users.

[0005] On one hand, this application provides an image processing method, comprising: extracting features from a target sample image using a first image encoder and a second image encoder of an image segmentation model to obtain a first image feature and a second image feature; determining a target embedding feature of the target sample image based on the first image feature and the second image feature; wherein the target embedding feature is obtained by fusing a first attention feature and a second attention feature, and the first attention feature and the second attention feature are determined based on the first image feature and the second image feature; performing attention processing on the target embedding feature, a reference embedding feature of a reference image, and a mask embedding feature of a mask image corresponding to the reference image to obtain aligned embedding features; training the image segmentation model based on the aligned embedding features, and performing image processing on the image to be segmented based on the trained image segmentation model.

[0006] In one embodiment, determining the target embedding feature of the target sample image based on the first image feature and the second image feature includes: performing gated attention fusion processing on the first image feature and the second image feature to obtain the first attention feature and the second attention feature; performing an addition operation on the first attention feature and the second attention feature to obtain a fused feature; and using the fused feature as the target embedding feature of the target sample image.

[0007] In one embodiment, the step of performing gated attention fusion processing on the first image features and the second image features to obtain the first attention features and the second attention features includes: stitching the first image features and the second image features together to obtain stitched features; performing channel attention fusion processing on the stitched features to obtain channel-level stitched features; and performing spatial attention fusion processing on the channel-level stitched features to obtain the first attention features or the second attention features.

[0008] In one embodiment, the step of performing channel attention fusion processing on the splicing features to obtain channel-level splicing features includes: performing global pooling processing on the splicing features to obtain global features in the channel dimension; determining channel-level attention weights based on the global features; and determining the channel-level splicing features based on the attention weights.

[0009] In one embodiment, the step of performing spatial attention fusion processing on the channel-level stitched features to obtain the first attention feature or the second attention feature includes: performing convolution processing on the channel-level stitched features to obtain gating weights; determining a first sub-attention feature of the first image feature based on the gating weights; determining a second sub-attention feature of the second image based on a preset value and the gating weights; and performing an addition operation on the first sub-attention feature and the second sub-attention feature to obtain the first attention feature or the second attention feature.

[0010] In one embodiment, before extracting features from the target sample image using the first and second image encoders of the image segmentation model to obtain the first and second image features, the method further includes: extracting features from the reference image using the third and fourth image encoders of a dual-path model to obtain the third and fourth image features; determining reference embedding features of the reference image based on the third and fourth image features using the dual-path model; wherein the reference embedding features are obtained by fusing the third and fourth attention features, and the third and fourth attention features are determined based on the third and fourth image features; and the first cue encoder of the dual-path model... The input sampling points, sampling boxes, sampling mask images, and sampling text are used to extract features to obtain a first cue feature; and the second cue encoder of the dual-path model is used to extract features from the input sampling points, sampling boxes, and sampling mask images to obtain a second cue feature; the first decoder of the dual-path model decodes the third attention feature and the first cue feature to obtain a first decoded feature; and the second decoder of the dual-path model decodes the fourth attention feature and the second cue feature to obtain a second decoded feature; the dual-path model is trained based on the first decoded feature and the second decoded feature, and other reference images are processed based on the trained dual-path model to obtain reference embedding features of the other reference images.

[0011] In one embodiment, before performing attention processing on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image to obtain aligned embedding features, the method further includes: obtaining the reference embedding feature of the reference image obtained by processing the reference image through the dual-path model; after performing attention processing on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image to obtain aligned embedding features, the method further includes: decoding the aligned embedding features through each decoder in the image segmentation model to obtain a first decoded feature and a second decoded feature; training the image segmentation model based on the aligned embedding features, and performing image processing on the image to be segmented based on the trained image segmentation model, includes: training the image segmentation model based on the first decoded feature and the second decoded feature, and performing image processing on the image to be segmented based on the trained image segmentation model.

[0012] In one embodiment, training the image segmentation model based on the first decoding feature and the second decoding feature includes: determining local output probability values ​​and global output probability values ​​based on the target sample image, the first decoding feature, and the second decoding feature, respectively, until the loss value determined based on the local output probability value and the global output probability value satisfies a preset loss condition, so as to obtain the trained image segmentation model.

[0013] On one hand, this application also provides an image processing apparatus, comprising: an extraction module, configured to extract features from a target sample image using a first image encoder and a second image encoder of an image segmentation model, respectively, to obtain a first image feature and a second image feature; a determination module, configured to determine a target embedding feature of the target sample image based on the first image feature and the second image feature; wherein the target embedding feature is obtained by fusing a first attention feature and a second attention feature, and the first attention feature and the second attention feature are determined based on the first image feature and the second image feature; a processing module, configured to perform attention processing on the target embedding feature, a reference embedding feature of a reference image, and a mask embedding feature of a mask image corresponding to the reference image, to obtain aligned embedding features; and a training module, configured to train the image segmentation model based on the aligned embedding features, and perform image processing on the image to be segmented based on the trained image segmentation model.

[0014] On one hand, this application also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: extracting features from a target sample image using a first image encoder and a second image encoder of an image segmentation model to obtain a first image feature and a second image feature; determining a target embedding feature of the target sample image based on the first image feature and the second image feature; wherein the target embedding feature is obtained by fusing a first attention feature and a second attention feature, and the first attention feature and the second attention feature are determined based on the first image feature and the second image feature; performing attention processing on the target embedding feature, a reference embedding feature of a reference image, and a mask embedding feature of a mask image corresponding to the reference image to obtain aligned embedding features; training the image segmentation model based on the aligned embedding features, and performing image processing on the image to be segmented based on the trained image segmentation model.

[0015] On one hand, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the following steps: extracting features from a target sample image using a first image encoder and a second image encoder of an image segmentation model to obtain a first image feature and a second image feature; determining a target embedding feature of the target sample image based on the first image feature and the second image feature; wherein the target embedding feature is obtained by fusing a first attention feature and a second attention feature, and the first attention feature and the second attention feature are determined based on the first image feature and the second image feature; performing attention processing on the target embedding feature, a reference embedding feature of a reference image, and a mask embedding feature of a mask image corresponding to the reference image to obtain aligned embedding features; training the image segmentation model based on the aligned embedding features, and performing image processing on the image to be segmented based on the trained image segmentation model.

[0016] On one hand, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps: extracting features from a target sample image using a first image encoder and a second image encoder of an image segmentation model to obtain a first image feature and a second image feature; determining a target embedding feature of the target sample image based on the first image feature and the second image feature; wherein the target embedding feature is obtained by fusing a first attention feature and a second attention feature, and the first attention feature and the second attention feature are determined based on the first image feature and the second image feature; performing attention processing on the target embedding feature, a reference embedding feature of a reference image, and a mask embedding feature of a mask image corresponding to the reference image to obtain aligned embedding features; training the image segmentation model based on the aligned embedding features, and performing image processing on the image to be segmented based on the trained image segmentation model.

[0017] The aforementioned image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product extract features from a target sample image using a first image encoder and a second image encoder of an image segmentation model, respectively, to obtain first image features and second image features. Based on the first image features and second image features, a target embedding feature of the target sample image is determined. The target embedding feature is obtained by fusing a first attention feature and a second attention feature, which are determined based on the first image features and the second image features. Further, attention processing is performed on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image to obtain aligned embedding features. Based on the aligned embedding features, the image segmentation model is trained, and based on the trained image segmentation model, image processing is performed on the image to be segmented. Since the target embedding feature is obtained by fusing the first attention feature and the second attention feature, and both the first and second attention features are determined based on the first image feature and the second image feature, it can adaptively adjust the information contribution of the first image feature and the second image feature extracted from the target sample image. This achieves the purpose and function of strengthening important features and suppressing redundant information, thereby effectively improving the expressive power of the fused feature, i.e., the target embedding feature. The fused target embedding feature considers both global information and retains local details. As a result, the aligned embedding feature determined based on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image is also more accurate. This enables the trained image segmentation model, which is trained based on the aligned embedding feature, to perform high-precision image segmentation processing on the image to be segmented. This effectively improves the accuracy of image segmentation and achieves real-time and high-precision image segmentation. It effectively solves the problem that traditional methods cannot perform fine segmentation of wrinkles and fine lines in the skin measurement field, thereby effectively improving the visual effect of image segmentation processing and enhancing the user experience, bringing convenience to users. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0020] Figure 2This is a flowchart illustrating an image processing method in one embodiment;

[0021] Figure 3 This is a schematic diagram of the target graph network structure in one embodiment;

[0022] Figure 4 This is a schematic diagram of the sampling process during the preprocessing stage of an image processing method provided in one embodiment;

[0023] Figure 5 This is a schematic diagram of the gating attention fusion process in one embodiment;

[0024] Figure 6 This is a schematic diagram of a network structure for processing a reference diagram, provided in one embodiment.

[0025] Figure 7 This is a flowchart illustrating the internal process of the prompt encoding module in a dual-path model provided in one embodiment;

[0026] Figure 8 This is a schematic diagram of the internal process of the decoder module 1 in an image segmentation model provided in one embodiment;

[0027] Figure 9 This is a schematic diagram of the internal process of the decoder module 2 in an image segmentation model provided in one embodiment;

[0028] Figure 10 This is a structural block diagram of an image processing device in one embodiment;

[0029] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0030] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] The image processing method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server. Server 104 can be a backend server for an image application. After terminal 102 acquires the image to be segmented, it can send the acquired image to be segmented to the backend server of the image application, i.e., server 104, so that server 104 can extract features from the target sample image through the first image encoder and the second image encoder of the image segmentation model, respectively, to obtain the first image features and the second image features, and determine the target embedding features of the target sample image based on the first image features and the second image features. The target embedding features are obtained by fusing the first attention features of the first image features and the second attention features of the second image features, and the first attention features and the second attention features are determined according to the gating weights. Further, server 104 can perform attention processing on the target embedding features, the reference embedding features of the reference image, and the mask embedding features of the mask image corresponding to the reference image to obtain aligned embedding features. The image segmentation model is trained based on the aligned embedding features, and the image to be segmented is processed based on the trained image segmentation model to obtain the image segmentation processing result. Further, server 104 can return the image segmentation processing result to terminal 102 so that terminal 102 can display the image segmentation processing result after image processing of the image to be segmented.

[0032] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0033] In one exemplary embodiment, such as Figure 2 As shown, an image processing method is provided, which is applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps 202 to 208. Wherein:

[0034] Step 202: The first image encoder and the second image encoder of the image segmentation model are used to extract features from the target sample image to obtain the first image features and the second image features.

[0035] Image segmentation model refers to a model used to segment different regions in an image. For example, the image segmentation model in this application can be a dual-path training branch model composed of two encoders, commonly known as a Siamese network. The dual-path model in this application is mainly used to improve the model's discriminative ability, that is, it can adaptively adjust the information contribution of the first image feature and the second image feature extracted from the same target sample image, so as to strengthen important features and suppress redundant information, thereby effectively improving the expressive ability of the fused feature, i.e. the target embedding feature. This makes the fused target embedding feature consider both global information and retain local details, which helps to improve the model's discriminative ability.

[0036] The first encoder refers to one encoder in a dual-channel model, for example, such as... Figure 3 The diagram shown is a schematic of the target graph network structure. The first encoder can be as follows: Figure 3 The image encoding module 1 in the dual-path model shown.

[0037] The second encoder refers to another encoder in a dual-channel model. For example, the second encoder can be such as... Figure 3 Image encoding module 2 in the dual-path model shown.

[0038] The target sample image refers to the training sample data used during the training phase. For example, the target sample image in this application can be such as... Figure 3 The image data shown may include multiple target sample images.

[0039] The first and second image features in this application are only used to distinguish the features extracted by different encoders in the dual-path model. For example, the first image feature in this application can be as follows: Figure 3 The image encoding module 1 shown is the image encoding feature extracted by the first encoder from the target sample image. The second image feature can be as follows: Figure 3 The image encoding module 2 shown is the image encoding feature extracted from the target sample image by the second encoder.

[0040] Step 204: Determine the target embedding features of the target sample image based on the first image features and the second image features; wherein, the target embedding features are obtained by fusing the first attention features and the second attention features, and the first attention features and the second attention features are determined based on the first image features and the second image features.

[0041] Here, the target embedding feature refers to the embedding feature corresponding to the target sample image determined based on the first image feature and the second image feature, used to characterize the information contained in the target sample image. For example, the target embedding feature in this application can be as follows: Figure 3 As shown: The fused feature obtained by performing a "+" operation on the gated attention feature output by the gated attention fusion module 1 after processing the first image feature (i.e., the first attention feature) and the gated attention feature output by the gated attention fusion module 2 after processing the second image feature (i.e., the second attention feature) is the target embedding feature.

[0042] The first and second attention features in this application are only used to distinguish the attention features corresponding to image features extracted by different encoders in the dual-path model. For example, the first attention feature in this application refers to: ... Figure 3 The first image features extracted from the target sample image by image encoding module 1 (i.e., the first encoder) and the second image features extracted from the target sample image by image encoding module 2 (i.e., the second encoder) are input together into gated attention fusion module 1. After processing the first and second image features, the gated attention features output by gated attention fusion module 1 are the first attention features. The second attention features in this application refer to: ... Figure 3 The second image features extracted from the target sample image by the image encoding module 2 (i.e., the second encoder) and the first image features extracted from the target sample image by the image encoding module 1 (i.e., the first encoder) are input together into the gating attention fusion module 2. After the gating attention fusion module 2 processes the second image features and the first image features, the output gating attention features are the second attention features.

[0043] Gating weights refer to the weights generated during gating attention fusion processing of the first and second image features, which are used to control the fusion of the two features (i.e., to control the weights of the first and second image features in the input).

[0044] Optionally / exemplarily, devices used by different users (operation objects) can interact with the image application (or image generation system). When a user (operation object) wants to perform facial detail segmentation on a personalized image (such as a personalized face image), the user can open the image application (APP) on the terminal by triggering an operation, and enter the image generation page of the image application by selecting an operation. That is, the user can log in to the image application (such as an image beautification application) by triggering an operation. Furthermore, the user can trigger an information input operation or an information selection operation on the image application's generation page displayed on the terminal, so that the terminal responds to the above-mentioned information input operation or information selection operation triggered by the user and obtains the image to be segmented by the user. Furthermore, the terminal responds to the image recognition request (such as a request for segmenting the details and textures contained in the image) triggered by the operation object on the image application's generation page, calls the trained image segmentation model, and performs image processing on the image to be segmented by the trained image segmentation model to obtain the image segmentation result, and then visualizes the image segmentation result.

[0045] In training an image segmentation model for image (segmentation) processing of the image to be segmented, the terminal can acquire target sample images containing different sample target objects (such as different faces), and train the initial image segmentation model based on the target sample images to obtain the trained image segmentation model. That is, the image segmentation model in this application can be a neural network model pre-trained through two-stage neural network learning, capable of performing refined segmentation of skin-related areas such as wrinkles and fine lines in the image to be segmented.

[0046] It is understood that the method provided in this application can be implemented through interaction between the terminal and the backend server of the image application, or through interaction between the frontend and backend of the terminal. That is, the frontend of the terminal is used to display the image to be segmented and the image segmentation result of image segmentation processing of the image to be segmented based on the trained image segmentation model, and the backend of the terminal is equivalent to the backend server, which is used to train the initial image segmentation model and call the trained image segmentation model to perform image segmentation and other logical processing.

[0047] Specifically, during the training phase, the terminal can extract features from the target sample image using the first and second image encoders of the initial image segmentation model, respectively, to obtain first image features and second image features. Based on these first and second image features, the target embedding features of the target sample image are then determined. For example, such as... Figure 3The diagram shown illustrates the network structure of the image segmentation model invoked by the image processing method provided in this application. The example used is a scenario where fine-grained segmentation of skin-related features contained in the image to be segmented is employed. During the training phase, the terminal can... Figure 3 The image segmentation model shown includes image encoding module 1 (the first image encoder) and image encoding module 2 (the second image encoder) for processing the target sample image (i.e., ...). Figure 3 The terminal extracts features from the image data (in the image data), outputting the first image features and the second image features; furthermore, the terminal can... Figure 3 The first image feature output by image encoding module 1 and the second image feature output by image encoding module 2 in the image segmentation model shown are used together as input parameters of gated attention fusion module 1. After the gated attention fusion module 1 processes the first and second image features, it outputs the first attention feature; and the terminal can then use the first image feature as shown in the image segmentation model. Figure 3 The second image feature output by image encoding module 2 in the image segmentation model shown, together with the first image feature output by image encoding module 1, is used as the input parameter of gated attention fusion module 2. After the gated attention fusion module 2 processes the first and second image features, it outputs the second attention feature. Furthermore, the terminal can... Figure 3 The first attention feature output by the gated attention fusion module 1 and the second attention feature output by the gated attention fusion module 2 are added together to obtain the fused feature, which is then used as the target embedding feature of the target sample image.

[0048] Among them, in the process of such Figure 3 In the image segmentation model shown, the gated attention fusion module 1 processes the first image feature and the second image feature, generating gate weights to control the fusion of the two features (i.e., to control the weights of the input first and second image features). The gated attention fusion module 1 processes the first and second image features according to the generated gate weights, outputting the first attention feature. For example, the gated attention fusion module 1 determines the first sub-attention feature of the first image feature and the second sub-attention feature of the second image feature based on the gate weights, and then adds the first and second sub-attention features to obtain the first attention feature. Similarly, by using... Figure 3In the process of processing the first image feature and the second image feature, the gated attention fusion module 2 of the image segmentation model shown also generates gate weights to control the fusion of the two features (i.e., to control the weights of the input first image feature and the second image feature). That is, the gated attention fusion module 2 can process the first image feature and the second image feature according to the generated gate weights and output the second attention feature. For example, based on the gate weights, the gated attention fusion module 2 determines the first sub-attention feature of the first image feature and the second sub-attention feature of the second image feature, and then performs an addition operation on the first sub-attention feature and the second sub-attention feature to obtain the second attention feature.

[0049] Step 206: Attention processing is performed on the target embedding features, the reference embedding features of the reference image, and the mask embedding features of the mask image corresponding to the reference image to obtain the aligned embedding features.

[0050] The reference embedding features of the reference image can be embedding features obtained by processing the reference image through other dual-path models. For example, the reference embedding features in this application can be the reference embedding features of the reference image obtained during the first stage of training (i.e., training the reference graph network model based on the reference image). Currently, training the target graph network model based on the target (sample) image can be regarded as the second stage of training. That is, the training method used in the embodiments of this application can be a method based on the co-training of the reference graph and the target graph.

[0051] The mask image corresponding to the reference image refers to the mask image (i.e., the reference image mask) that is determined in advance during the sampling stage. For example, such as Figure 4 The diagram shown illustrates the sampling process of the image processing method provided in this application during the preprocessing stage. The mask image corresponding to the reference image in this application can be as follows: Figure 4 The mask shown is a portion of the masks selected from the multiple masks corresponding to "one skin measurement image" and used as "MasksRef", which is the mask image corresponding to the reference image.

[0052] Mask embedding features refer to features used to characterize information in a masked image. For example, the mask embedding features in this application can be achieved through methods such as... Figure 3 The feature obtained by the memory encoding module of the (initial) image segmentation model shown in the figure after encoding the mask image (i.e., "MasksRef") corresponding to the input reference image is the mask embedding feature of "MasksRef".

[0053] Attention processing can be achieved through methods such as Figure 3The memory attention module implementation of the (initial) image segmentation model shown in the figure uses the target embedding features, the reference embedding features of the reference image, and the mask embedding features of the mask image corresponding to the reference image as... Figure 3 The input parameters of the memory attention module of the (initial) image segmentation model shown are processed by the memory attention module, and the output is the aligned embedded features.

[0054] Step 208: Train the image segmentation model based on the aligned embedding features, and perform image processing on the image to be segmented based on the trained image segmentation model.

[0055] Specifically, during the training phase, the terminal extracts features from the target sample image using the first and second image encoders of the (initial) image segmentation model, obtaining first and second image features respectively. Based on these first and second image features, the terminal determines the target embedding features of the target sample image. Then, the terminal can obtain the reference embedding features of the reference image obtained in the previous training phase (training the reference graph network model based on the reference image), as well as the pre-processed mask image corresponding to the reference image. The feature obtained after encoding the input mask image (i.e., "MasksRef") corresponding to the reference image using the memory encoding module in the (initial) image segmentation model is the mask embedding feature of "MasksRef". The initial image segmentation model performs memory attention processing on the target embedding features, the reference embedding features of the reference image, and the mask embedding features of "MasksRef" to obtain aligned embedding features. Furthermore, the terminal can decode the aligned embedding features through the decoders in the image segmentation model (such as the first decoder and the second decoder) to obtain the first decoded features and the second decoded features. The (initial) image segmentation model is then trained based on the first decoded features and the second decoded features until the loss value determined based on the target sample image, the first decoded features, and the second decoded features meets the preset loss condition. This results in the trained image segmentation model, which is then used to process the image to be segmented.

[0056] For example, let's illustrate this with a scenario where fine-grained segmentation is performed on skin-related features contained in an image to be segmented. During the training phase, the terminal uses methods such as... Figure 3 The image encoding module 1 (i.e., the first image encoder) and image encoding module 2 (i.e., the second image encoder) of the (initial) image segmentation model shown in the figure respectively encode the target sample image (i.e., Figure 3After extracting features from the image data (in the image), obtaining the first image features and the second image features, and determining the target embedding features of the target sample image based on the first image features and the second image features, the terminal can obtain the reference embedding features of the reference image obtained in the previous stage of training (training the reference graph network model based on the reference image), i.e., ... Figure 3 The ImageEmbedRef shown, and the mask image corresponding to the preprocessed reference image (i.e., MasksRef), are obtained by encoding the mask image (i.e., "MasksRef") corresponding to the input reference image through the memory encoding module in the (initial) image segmentation model. The resulting feature is the mask embedding feature of "MasksRef". This feature is then used as follows: Figure 3 The memory attention module of the (initial) image segmentation model shown performs memory attention processing on the target embedding features, the reference embedding features of the reference image (i.e., ImageEmbedRef), and the mask embedding features of "MasksRef" to obtain the aligned embedding features; furthermore, the terminal can use methods such as... Figure 3 In the image segmentation model shown, decoder module 1 and decoder module 2 decode the aligned embedded features to obtain the first decoded feature, Output1, and the second decoded feature, Output2. The (initial) image segmentation model is then trained based on these first and second decoded features until a loss value (e.g., based on the target sample image (input), the first decoded feature (Output1), and the second decoded feature (Output2) is reached. Training stops when the preset loss condition is met, and the trained image segmentation model is obtained. Then, image processing is performed on the image to be segmented based on the trained image segmentation model.

[0057] In this embodiment, the first image encoder and the second image encoder of the image segmentation model extract features from the target sample image to obtain the first image features and the second image features, and determine the target embedding features of the target sample image based on the first image features and the second image features. The target embedding features are obtained by fusing the first attention features and the second attention features, which are determined based on the first image features and the second image features. Further, attention processing is performed on the target embedding features, the reference embedding features of the reference image, and the mask embedding features of the mask image corresponding to the reference image to obtain aligned embedding features. The image segmentation model is trained based on the aligned embedding features, and image processing is performed on the image to be segmented based on the trained image segmentation model. Since the target embedding feature is obtained by fusing the first attention feature and the second attention feature, and both the first and second attention features are determined based on the first image feature and the second image feature, it can adaptively adjust the information contribution of the first image feature and the second image feature extracted from the target sample image. This achieves the purpose and function of strengthening important features and suppressing redundant information, thereby effectively improving the expressive power of the fused feature, i.e., the target embedding feature. The fused target embedding feature considers both global information and retains local details. As a result, the aligned embedding feature determined based on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image is also more accurate. This enables the trained image segmentation model, which is trained based on the aligned embedding feature, to perform high-precision image segmentation processing on the image to be segmented. This effectively improves the accuracy of image segmentation and achieves real-time and high-precision image segmentation. It effectively solves the problem that traditional methods cannot perform fine segmentation of wrinkles and fine lines in the skin measurement field, thereby effectively improving the visual effect of image segmentation processing and enhancing the user experience, bringing convenience to users.

[0058] In an exemplary embodiment, the step of determining the target embedding features of a target sample image based on first image features and second image features includes:

[0059] Gated attention fusion processing is performed on the first image features and the second image features to obtain the first attention features and the second attention features;

[0060] The first attention feature and the second attention feature are added together to obtain the fused feature;

[0061] The fused features are used as the target embedding features of the target sample image.

[0062] Among them, gated attention fusion processing is used to weight and fuse multiple features through an attention mechanism to improve feature representation capabilities. For example, the gated attention fusion processing in this application may include two sub-processing steps: one is to perform channel attention fusion processing on multiple features, for example, through the channel attention fusion module (ChannelAttnFusion): this module performs global pooling on multiple features (such as concatenated features) to obtain global features in the channel dimension, and then performs convolution (Conv), GELU activation function, convolution (Conv) and Sigmoid activation to generate channel-level attention weights, and broadcasts the generated channel-level attention weights to the spatial dimension of the original input for weighted feature fusion to obtain the features after channel attention processing (also known as channel-level features). Secondly, spatial attention fusion is performed on multiple features. For example, the weighted features from the previous step are fed into the spatial attention fusion module (SpatialAttnFusion) as input parameters. This allows the module to further process the weighted features through convolution (Conv), layer normalization (LN), GELU, convolution (Conv), and sigmoid activation, generating the final gate weights (Gated). These gate weights control the weights for fusing the two feature paths. For instance, Gated * Feats1 and (1 - Gated) * Feats2 are calculated, which adaptively adjusts the contributions of the two feature paths using the gate weights. Finally, an addition operation is used to fuse the two feature sets, resulting in the final output feature, either the first attention feature or the second attention feature. That is, the gated attention fusion processing module in this application embodiment can adaptively adjust the information contribution of Feats1 and Feats2 to strengthen important features and suppress redundant information, thereby improving the expressive power of the fused features (i.e., the first attention feature or the second attention feature). It can be understood that the combination of channel attention and spatial attention designed in this application embodiment makes the fusion process consider both global information and retain local details, which helps to improve the model's discrimination ability.

[0063] Specifically, let's take the scenario of performing fine-grained segmentation of skin-related features contained in the image to be segmented as an example. During the training phase, the terminal uses methods such as... Figure 3 The image encoding module 1 (i.e., the first image encoder) and image encoding module 2 (i.e., the second image encoder) of the (initial) image segmentation model shown in the figure respectively encode the target sample image (i.e., Figure 3After extracting features from the image data (in the image), obtaining first image features and second image features, the terminal can perform gated attention fusion processing on the first image features and second image features to obtain first attention features and second attention features. For example, the terminal can perform such... Figure 3 The first image feature output by image encoding module 1 and the second image feature output by image encoding module 2 in the image segmentation model shown are used together as input parameters of gated attention fusion module 1. After the gated attention fusion module 1 processes the first and second image features, it outputs the first attention feature; and the terminal can then use the first image feature as shown in the image segmentation model. Figure 3 The second image feature output by image encoding module 2 in the image segmentation model shown, together with the first image feature output by image encoding module 1, is used as the input parameter of gated attention fusion module 2. After the gated attention fusion module 2 processes the first and second image features, it outputs the second attention feature. Furthermore, the terminal can... Figure 3 The first attention feature output by the gated attention fusion module 1 and the second attention feature output by the gated attention fusion module 2 are combined with a "+" operation to obtain the fused feature, which is then used as the target embedding feature of the target sample image.

[0064] In this embodiment, a gated attention fusion processing mechanism is introduced. This mechanism can adaptively adjust the information contribution of Feats1 and Feats2 to strengthen important features and suppress redundant information, thereby improving the expressive power of the fused features (i.e., the first attention feature or the second attention feature). That is, the channel attention and spatial attention combined in this embodiment make the fusion process consider both global information and local details, which helps to improve the model's discrimination ability and thus effectively improves the accuracy of the trained segmentation model in segmenting images.

[0065] In an exemplary embodiment, the step of performing gated attention fusion processing on the first image features and the second image features to obtain the first attention features and the second attention features includes:

[0066] The first image features and the second image features are concatenated to obtain the concatenated features;

[0067] Channel attention fusion processing is performed on the splicing features to obtain channel-level splicing features;

[0068] Spatial attention fusion processing is performed on the channel-level splicing features to obtain either the first attention feature or the second attention feature.

[0069] Specifically, let's take the scenario of performing fine-grained segmentation of skin-related features contained in the image to be segmented as an example. During the training phase, the terminal uses methods such as... Figure 3 The image encoding module 1 (i.e., the first image encoder) and image encoding module 2 (i.e., the second image encoder) of the (initial) image segmentation model shown in the figure respectively encode the target sample image (i.e., Figure 3 After extracting features from the image data (in the image), obtaining first image features and second image features, the terminal can perform gated attention fusion processing on the first image features and second image features to obtain first attention features and second attention features. For example, the terminal can perform such... Figure 3 The first image feature output by image encoding module 1 and the second image feature output by image encoding module 2 in the image segmentation model shown are used together as input parameters of gated attention fusion module 1. After the gated attention fusion module 1 processes the first and second image features, it outputs the first attention feature; and the terminal can then use the first image feature as shown in the image segmentation model. Figure 3 The second image feature output by the image encoding module 2 in the image segmentation model shown, together with the first image feature output by the image encoding module 1, is used as the input parameter of the gated attention fusion module 2. After the gated attention fusion module 2 processes the first image feature and the second image feature, the second attention feature is output.

[0070] In such Figure 3 When the gated attention fusion module 1 or gated attention fusion module 2 shown processes the first image features and the second image features, such as... Figure 5 The diagram shown illustrates the process of gated attention fusion processing. Assume the first image feature is... Figure 5 Feats1 and the second image feature are Figure 5 In Feats2, when the terminal processes the first image features and the second image features through gated attention fusion module 1 or gated attention fusion module 2 in the image segmentation model, such as... Figure 5 As shown, gated attention fusion module 1 or gated attention fusion module 2 can first concatenate the first image feature Feats1 and the second image feature Feats2 to obtain concatenated feature Feats12, and then perform channel attention fusion processing on the concatenated feature Feats12 to obtain channel-level concatenated features. Finally, spatial attention fusion processing is performed on the channel-level concatenated features to obtain the desired result. Figure 5The final output shown is the Output (first attention feature or second attention feature). This allows the fusion process, which combines channel attention and spatial attention as designed in this embodiment, to consider both global information and preserve local details, thus improving the model's discriminative ability and effectively enhancing the accuracy of the trained segmentation model in image segmentation.

[0071] In one exemplary embodiment, the step of performing channel attention fusion processing on the stitched features to obtain channel-level stitched features includes:

[0072] Global pooling is performed on the spliced ​​features to obtain global features in the channel dimension;

[0073] Channel-level attention weights are determined based on global features;

[0074] Channel-level splicing features are determined based on attention weights.

[0075] Specifically, let's take the scenario of performing fine-grained segmentation of skin-related features contained in the image to be segmented as an example. During the training phase, assume the first image feature is... Figure 5 Feats1 and the second image feature are Figure 5 Feats2 in the example, the terminal passes through, for example, Figure 3 When the gated attention fusion module 1 or gated attention fusion module 2 in the image segmentation model shown processes the first image features and the second image features, such as... Figure 5 As shown, gated attention fusion module 1 or gated attention fusion module 2 can first concatenate the first image feature Feats1 and the second image feature Feats2 to obtain concatenated feature Feats12, and then perform channel attention fusion processing on the concatenated feature Feats12 to obtain channel-level concatenated features. Finally, spatial attention fusion processing is performed on the channel-level concatenated features to obtain the desired result. Figure 5 The final output shown is the Output (first attention feature or second attention feature). This allows the fusion process, which combines channel attention and spatial attention as designed in this embodiment, to consider both global information and preserve local details, thus improving the model's discriminative ability and effectively enhancing the accuracy of the trained segmentation model in image segmentation.

[0076] In one exemplary embodiment, the step of performing spatial attention fusion processing on channel-level stitching features to obtain a first attention feature or a second attention feature includes:

[0077] Convolutional processing is performed on the channel-level splicing features to obtain the gating weights;

[0078] The first sub-attention feature is determined based on the gating weights to identify the first image feature;

[0079] The second sub-attention features of the second image are determined based on preset values ​​and gating weights;

[0080] Add the first sub-attention feature and the second sub-attention feature to obtain the first attention feature or the second attention feature.

[0081] The preset value can be 1.

[0082] Specifically, let's take the scenario of refining the segmentation of skin-related features in an image to be segmented as an example. During the training phase, assume the first image feature is... Figure 5 Feats1 and the second image feature are Figure 5 Feats2 in the example, the terminal passes through, for example, Figure 3 When the gated attention fusion module 1 or gated attention fusion module 2 in the image segmentation model shown processes the first image features and the second image features, such as... Figure 5As shown, the gated attention fusion module 1 or the gated attention fusion module 2 can first concatenate the first image feature Feats1 and the second image feature Feats2 to obtain the concatenated feature Feats12, and then perform channel attention fusion processing on the concatenated feature Feats12 to obtain the channel-level concatenated feature. After that, the terminal can use the gated attention fusion module 1 or the gated attention fusion module 2 in the image segmentation model to perform convolution (Conv), layer normalization (LN), GELU, convolution (Conv) and sigmoid activation on the channel-level concatenated feature to generate the final gate weight Gated. The gate weight Gated is used to control the weight of the two-way feature fusion. Furthermore, gated attention fusion module 1 or gated attention fusion module 2 can determine the first sub-attention feature of the first image feature based on the gate weight, i.e., calculate Gated * Feats1 = the first sub-attention feature, and determine the second sub-attention feature of the second image based on the preset value and gate weight, i.e., calculate (1 - Gated) * Feats2 = the second sub-attention feature. That is, the contribution of the two features is adaptively adjusted by using the gate weight, and the two parts of features, namely Gated * Feats1 and (1 - Gated) * Feats2, are fused by an addition operation to obtain the final output feature, i.e., the first attention feature or the second attention feature. This allows for adaptive adjustment of the information contribution of Feats1 and Feats2 to strengthen important features and suppress redundant information, thereby improving the expressive power of the fused feature (i.e., the first attention feature or the second attention feature). That is, the combination of channel attention and spatial attention designed in this embodiment makes the fusion process consider both global information and retain local details, which helps to improve the model's discrimination ability and makes the model's image segmentation results more accurate.

[0083] In one exemplary embodiment, before extracting features from the target sample image using a first image encoder and a second image encoder of an image segmentation model to obtain the first image features and the second image features, the method further includes:

[0084] The third and fourth image encoders of the dual-path model are used to extract features from the reference image to obtain the third image features and the fourth image features.

[0085] Using a dual-path model, reference embedding features of a reference image are determined based on third and fourth image features. The reference embedding features are obtained by fusing third and fourth attention features, which are determined based on third and fourth image features.

[0086] The first prompt encoder of the dual-path model extracts features from the input sampling points, sampling boxes, sampling mask images, and sampling text to obtain the first prompt features; and the second prompt encoder of the dual-path model extracts features from the input sampling points, sampling boxes, and sampling mask images to obtain the second prompt features.

[0087] The first decoder of the dual-path model decodes the third attention feature and the first cue feature to obtain the first decoded feature; and the second decoder of the dual-path model decodes the fourth attention feature and the second cue feature to obtain the second decoded feature.

[0088] The dual-path model is trained based on the first and second decoding features, and other reference images are processed based on the trained dual-path model to obtain the reference embedding features of other reference images.

[0089] In this embodiment, the dual-path model refers to a model trained based on a reference (sample) image. The trained dual-path model can be used to process the reference image, for example, as... Figure 6 The diagram shown is a schematic of the network structure provided in this application for processing a reference graph. The network result of the dual-path model in this embodiment can be... Figure 6 The network structure shown is shown in the figure.

[0090] The third encoder refers to one encoder in a dual-channel model. For example, the third encoder can be such as... Figure 6 The image encoding module 1 in the dual-path model shown.

[0091] The fourth encoder refers to another encoder in a dual-channel model; for example, the fourth encoder could be such as... Figure 6 Image encoding module 2 in the dual-path model shown.

[0092] A reference image refers to the training sample data used during the training phase (e.g., the first phase of training). For example, the reference image in this application could be such as... Figure 6 The image data shown may include multiple reference images.

[0093] The third and fourth image features in this application are only used to distinguish the features extracted by different encoders in the dual-path model. For example, the third image feature in this application can be as follows: Figure 6 The image encoding module 1 shown is the image encoding feature extracted from the reference image by the third encoder. The fourth image feature can be, for example, the image encoding feature extracted from the reference image by the third encoder. Figure 6 The image encoding module 2 shown is the image encoding feature extracted from the reference image by the fourth encoder.

[0094] Specifically, during the training phase, before the terminal extracts features from the target sample image using the first and second image encoders of the image segmentation model to obtain the first and second image features (i.e., before the second training phase), the terminal can first perform a first-stage training on the reference graph network structure (model) based on the reference image. That is, the terminal can perform training on the reference graph network structure (model) using, for example... Figure 6 The dual-path model shown in the diagram uses a third image encoder (image encoding module 1) and a fourth image encoder (image encoding module 2) to extract features from a reference image, obtaining third image features and fourth image features, respectively. Furthermore, the terminal can use the dual-path model to determine the reference embedding features of the reference image based on the third and fourth image features. The reference embedding features are obtained by fusing the third and fourth attention features, which are determined based on the third and fourth image features. That is, the terminal can perform gated attention fusion processing on the third and fourth image features to obtain first and second attention features. For example, the terminal can... Figure 6 The third image feature output by image encoding module 1 and the fourth image feature output by image encoding module 2 in the image segmentation model shown are used together as input parameters of gated attention fusion module 1. After the gated attention fusion module 1 processes the third and fourth image features, the third attention feature is output; and the terminal can then use the third image feature as shown in the image segmentation model. Figure 6 The fourth image feature output by the image encoding module 2 in the image segmentation model shown, together with the third image feature output by the image encoding module 1, serves as the input parameter of the gated attention fusion module 2. After the gated attention fusion module 2 processes the third and fourth image features, the fourth attention feature is output.

[0095] In one embodiment, as Figure 6 When the gated attention fusion module 1 or the gated attention fusion module 2 shown processes the third image feature and the fourth image feature, it is assumed that the third image feature is... Figure 5 Feats1 and the fourth image feature are Figure 5 Feats2 in the example, the terminal passes through, for example, Figure 6 When the gated attention fusion module 1 or gated attention fusion module 2 in the dual-path model shown processes the third image features and the fourth image features, such as... Figure 5As shown, gated attention fusion module 1 or gated attention fusion module 2 can first concatenate the third image feature Feats1 and the fourth image feature Feats2 to obtain concatenated feature Feats12, and then perform channel attention fusion processing on the concatenated feature Feats12 to obtain channel-level concatenated features. Finally, spatial attention fusion processing is performed on the channel-level concatenated features to obtain the desired result. Figure 5 The final output shown is the Output (third attention feature or fourth attention feature).

[0096] Furthermore, the terminal can be accessed through methods such as Figure 6 The first cue encoder (i.e., cue encoding module 1) of the dual-path model shown extracts features from the sampling points, sampling boxes, sampling mask images, and sampling text input through sampling module 1 to obtain the first cue features; and through methods such as... Figure 6 The second cue encoder (i.e., cue encoding module 2) of the dual-path model shown extracts features from the sampling points, sampling frames, and sampling mask images input through sampling module 2 to obtain the second cue features. For example, Figure 7 The diagram shown is a flowchart illustrating the internal process of the prompt encoding module in the dual-path model provided in this application. Figure 7 Solid lines represent learnable parameters, dashed lines represent parameter freeze, and bold solid lines represent inputs and outputs, as shown in this application. Figure 6 The internal processing structure of the first cue encoder (i.e., cue encoding module 1) in the dual-channel model shown can be as follows: Figure 7As shown in the figure, the first cue encoder (i.e. cue encoding module 1) in this embodiment can receive four inputs from the sampling module: text, points, boxes, and masks. The text is processed by the CLIP model. CLIP (Contrastive Language-Image Pretraining) is a powerful multimodal model proposed by OpenAI. It can understand the relationship between images and text. This scheme uses its text encoder to extract text features. The extracted features are processed by a linear projection layer (FC+LN) and transformed into the same embedding space (dimension, feature length) as other sparse cues (points, boxes). They are then concatenated and sent to the mask decoder. The point embedding module EmbedPoints, the box embedding module EmbedBoxes, and the mask embedding module EmbedMasks all inherit the cue encoder of SAM2. During training, the parameters of CLIP are frozen and not updated, while the other boxes are trainable parameters. In the text encoder, supervised training is performed on multiple similar texts. In inference, in order to make the text prompts more flexible (i.e., to allow users to prompt for non-training text words or sentences), this scheme will also use CLIP to extract the feature vectors of all training texts in advance, and then use similarity matching to find the feature vector that is closest to the user's text prompt as input.

[0097] Furthermore, the terminal can be accessed through methods such as Figure 6 The first decoder (i.e., decoder module 1) of the dual-path model shown decodes the third attention feature and the first cue feature to obtain the first decoded feature, Output1; and through methods such as... Figure 6 The second decoder (i.e., decoder module 2) of the dual-path model shown decodes the fourth attention feature and the second cue feature to obtain the second decoded feature, i.e., Output1. Furthermore, the terminal can use the first and second decoded features output by the model to perform operations such as... Figure 6 The dual-path model shown is trained, and other reference images are processed based on the trained dual-path model to quickly and accurately obtain the reference embedding features of other reference images. Thus, the combination of channel attention and spatial attention designed in this embodiment allows the fusion process to consider both global information and preserve local details, helping to improve the model's discriminative ability and effectively enhancing the accuracy of the trained dual-path model in processing reference images.

[0098] In one exemplary embodiment, before performing attention processing on the target embedding features, the reference embedding features of the reference image, and the mask embedding features of the mask image corresponding to the reference image to obtain the aligned embedding features, the method further includes:

[0099] Obtain the reference embedding features of the reference image obtained by processing the reference image through a dual-path model;

[0100] After performing attention processing on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image to obtain the aligned embedding features, the method further includes:

[0101] The aligned embedded features are decoded by each decoder in the image segmentation model to obtain the first decoded feature and the second decoded feature.

[0102] The step of training the image segmentation model based on the aligned embedding features, and then performing image processing on the image to be segmented based on the trained image segmentation model, includes:

[0103] The image segmentation model is trained based on the first and second decoding features, and the image to be segmented is processed based on the trained image segmentation model.

[0104] In this embodiment, the reference embedding features of the reference image obtained can be: in relation to, for example Figure 6 The ImageEmbedRef obtained during training (e.g., the first stage of training) of the dual-path model (reference graph network model) shown can be stored in the target path so that the reference embedding features of the reference image, i.e., ImageEmbedRef, can be directly obtained from the target path during the second stage of training.

[0105] Specifically, let's take the scenario of performing fine-grained segmentation of skin-related features contained in the image to be segmented as an example. During the training phase, the terminal uses methods such as... Figure 3 The image segmentation model shown in the diagram uses a first image encoder and a second image encoder to extract features from the target sample image, obtaining first image features and second image features respectively. After determining the target embedding features of the target sample image based on the first and second image features, the terminal can obtain the target embedding features of the target sample image. Figure 6 The ImageEmbedRef obtained during training (i.e., the first stage of training) of the dual-path model (reference graph network model) shown is the reference image embedding vector. Alternatively, the terminal can directly obtain the reference embedding features of the stored reference image, i.e., ImageEmbedRef, from the target path. Furthermore, the terminal can obtain the mask image corresponding to the pre-processed reference image, i.e., MasksRef, and use methods such as... Figure 3The memory encoding module in the image segmentation model shown encodes the mask image (i.e., "MasksRef") corresponding to the input reference image, and the resulting feature is the mask embedding feature of "MasksRef". This feature is then processed using methods such as... Figure 3 The memory attention module of the image segmentation model shown performs memory attention processing on the target embedding features, the reference embedding features of the reference image (i.e., ImageEmbedRef), and the mask embedding features of "MasksRef" to obtain the aligned embedding features. Furthermore, the terminal can... Figure 3 In the image segmentation model shown, decoder module 1 and decoder module 2 decode the aligned embedded features to obtain the first decoded feature, Output1, and the second decoded feature, Output2. The (initial) image segmentation model is then trained based on these first and second decoded features until a loss value (e.g., based on the target sample image (input), the first decoded feature (Output1), and the second decoded feature (Output2) is reached. Training stops when the preset loss condition is met, thus obtaining the trained image segmentation model. Image processing is then performed on the image to be segmented based on this model. For example, ... Figure 8 The diagram shown illustrates the internal processing flow of the decoder module 1 in the image segmentation model provided in this application. Specifically, the terminal can process data through the decoder module 1 as shown in the diagram. Figure 8 In the image segmentation model shown, decoder module 1 decodes the aligned embedded features to obtain the first decoded feature, Output1. Figure 9 The diagram shown illustrates the internal processing flow of the decoder module 2 in the image segmentation model provided in this application. Specifically, the terminal can process the data through the decoder module 2 as shown in the diagram. Figure 9 In the image segmentation model shown, decoder module 2 decodes the aligned embedded features to obtain the second decoded feature, Output2.

[0106] In this embodiment, the trained image segmentation model can perform high-precision image segmentation processing on the image to be segmented, effectively improving the accuracy of image segmentation and achieving real-time and high-precision image segmentation. This effectively solves the problem that traditional methods cannot perform fine segmentation of wrinkles and fine lines in the skin measurement field, thereby effectively improving the visual effect of image segmentation processing and enhancing the user experience, bringing convenience to users.

[0107] In one exemplary embodiment, the step of training the image segmentation model based on the first decoding feature and the second decoding feature includes:

[0108] The local output probability value and the global output probability value are determined based on the target sample image, the first decoding feature and the second decoding feature, respectively. Training stops when the loss value determined based on the local output probability value and the global output probability value meets the preset loss condition, so as to obtain the trained image segmentation model.

[0109] The local output probability value refers to the output probability value calculated when not all pixels of the input image are applied. For example, the output probability value calculated when using PointRend (point sampling rendering) input image is called local AAF1, which can be used... express.

[0110] The global output probability value refers to the output probability value calculated when all pixels of the input image are applied; this is called global AAF1, and can be used... express.

[0111] Specifically, the terminal can be accessed through methods such as... Figure 3 In the image segmentation model shown, decoder module 1 and decoder module 2 decode the aligned embedded features to obtain the first decoded feature, i.e., Output1, and the second decoded feature, i.e., Output2. The (initial) image segmentation model is trained based on the first and second decoded features. The local output probability value and the global output probability value are determined based on the target sample image (input), the first decoded feature, and the second decoded feature (output), respectively. Training stops when the loss value determined based on the local output probability value and the global output probability value meets the preset loss condition, so as to obtain the trained image segmentation model. In this application embodiment, during the training phase, an adaptive asymmetric F1 loss (AAF1) is proposed. The loss function of this scheme can be used not only for semantic segmentation but also for SAM of interactive segmentation, especially for open-world interactive segmentation in the skin measurement domain. To make it easier to understand the loss function designed in this scheme, we assume that in N×C×M×H×W, N represents the batch, such as N images, C represents the number of masks sampled from one image, M represents the number of masks output by the reference graph network structure (or the target graph network structure), and H and W represent the corresponding width and height.

[0112] in, This indicates the input field.

[0113] This represents the target, where M is obtained through broadcasting.

[0114] in, This indicates that sigmoid is activated.

[0115] First, the input is transformed using a sigmoid function, as shown in formula (1) below:

[0116] (1)

[0117] The calculation obtained in formula (1) The C and M channels of Y are first interchanged, and then flattened as shown in formulas (2) and (3) below:

[0118] (2)

[0119] (3)

[0120] The positive and negative examples of the F1-score are calculated as shown in the following formulas (4) and (5):

[0121] (4)

[0122] (5)

[0123] At this time, it was obtained , All dimensions are NM.

[0124] The asymmetric weights are calculated as shown in formulas (6) and (7) below:

[0125] (6)

[0126] (7)

[0127] in, , , All of these are hyperparameters, and the results obtained at this time , The dimensions are all NM×CHW, and the average of the last dimension is calculated as shown in the following formula (8):

[0128] (8)

[0129] in, The dimensions are all NM, and the final AAF1 loss is shown in the following formula (9):

[0130] (9)

[0131] in, The dimension is also NM.

[0132] It is understood that in this embodiment, when X applies input to all pixels, it is called global AAF1. This means that the hyperparameters are set to: =10、 =1、 =2; when X uses PointRend (point sampling rendering), it is called local AAF1, using This indicates that the hyperparameters are set to [value]. =2、 =4、 =2. PointRend can adaptively sample the uncertain regions of the mask and apply stronger supervision to these key regions. PointRend can be used for optimization of local uncertain regions. This scheme applies both global and local AAF1 and applies a coefficient of 2 to the global AAF1 to obtain the final loss value as shown in the following formula (10):

[0133] (10)

[0134] During training, the training method used in this application only backpropagates the channel with the minimum loss among the M channels (Note: M refers to M in the loss dimension NM of Equation 9). This scheme applies both global and local losses, performing backpropagation for both the global and local minimum losses. This design effectively guides multiple output channels of the model to learn different segmentation characteristics. Total loss function The final dimension is determined by the batch size N and the number of model-defined output masks. This means that the loss function allows each sample to independently select the optimal output channel, whereas traditional methods are based on mask-level optimization, which may introduce noise, leading to more complex and unstable training. The sample-level optimization designed in this approach allows each sub-network to focus more on specific scenarios during backpropagation. For example, high-precision skin measurement data differs significantly from everyday shooting data. This loss function allows high-precision skin measurement data to focus on learning the left half of the Siamese network, while SAM data focuses on learning the right half, with both working synergistically. When blurred cues appear, this loss function can smooth out noise and implicitly filter out more general features, thereby effectively improving the stability of model training.

[0135] This application also provides an application scenario in which the above-described image processing method is applied. The method provided in this application embodiment can be applied to various personalized video or image fine segmentation scenarios. The following uses a scenario where a user interacts with an image enhancement system as an example to illustrate the image processing method provided in this application embodiment.

[0136] SAM2 stands for "Segment Anything Model 2," an improved version of Meta's (formerly Facebook) Segment Anything Model (SAM). The release of SAM propelled technological advancements in image segmentation, fostered the development of related industries, and has been widely applied across various fields. Compared to SAM1, SAM2 offers significant improvements in model structure, training data volume, and inference speed. SAM2 also introduces video tracking capabilities, enabling visual segmentation of cues in any video clip. Both SAM1 and SAM2 can segment specified objects using positive cues, negative cues, bounding box cues, or mask cues. To eliminate ambiguity, SAM2 employs four output channels: when multiple cues are present, only the first channel outputs; when only a single cue is present, three output channels are used to resolve ambiguities related to the whole, sub-parts, and smallest sub-parts. During training, only the minimum loss is backpropagated. Despite SAM2's strong interactivity, the following issues remain:

[0137] 1) Inflexible and limited cues: When users want to specify different objects using multiple cues, SAM2's segmentation performance significantly decreases if output as a single channel; if output as an ambiguous channel, the decoder needs to run multiple times, resulting in poor efficiency, and the required instance segmentation map needs to be manually merged. SAM2 only supports positive cues, negative cues, box cues, and mask cues, lacking semantically meaningful Chinese / English text cues or doodle-style mask cues.

[0138] 2) Insufficient segmentation precision: Especially in the field of skin analysis, it is unable to segment precisely, particularly for wrinkles and fine lines.

[0139] 3) Poor fuzzy segmentation ability and insufficient semantic understanding: In the field of skin measurement, there may be many spots and acne on the face. SAM2 cannot segment fuzzy objects and cannot segment all related lesions with a few cues.

[0140] 4) Does not support reference map segmentation: SAM2 cannot segment the target map using several high-precision annotated reference maps.

[0141] 5) High memory usage: All SAM2 prompts are interactively sampled during network training, which greatly increases memory usage and training time.

[0142] To improve user experience and address the problems of traditional methods, this application proposes a skin segmentation method based on SAM2. Specifically, the technical solution provided in this application redesigns the system framework for skin segmentation based on SAM2 and solves many of the problems existing in SAM2.

[0143] 1) Supports a wide range of prompt types: Supports both Chinese and English text prompts to segment various dimensions of skin measurement (pigmentation, wrinkles, pores, etc.), and supports positive prompts, negative prompts, box prompts, doodle mask prompts, and fine mask prompts. These prompts are interactively sampled during data loading to save GPU memory. In the solution design, the first channel is only used when text prompts are involved; the other three channels will adaptively exhibit different levels of ambiguity depending on the prompt.

[0144] 2) In order to segment fuzzy objects and avoid catastrophic forgetting of the original segmentation ability of SAM2, a model fine-tuning method suitable for skin measurement was designed, and a gated attention fusion module was proposed to improve the segmentation refinement.

[0145] 3) It supports automatic segmentation of the target image using a small number of reference images as hints, without requiring additional manual intervention.

[0146] 4) A new loss function was redesigned to address the issues of segmentation precision and point ambiguity.

[0147] The skin segmentation method based on SAM2 proposed in this application includes the following specific steps:

[0148] I. Sampling Module

[0149] like Figure 4 As shown, the dataset for this solution includes data unique to skin testing and open-source SAM data. Each skin testing image is labeled with various sub-dimension masks (e.g., teeth, eyes, any dimension, local dimension, or global dimension in a facial image), such as forehead wrinkles, eye bags, pigmentation, eyes, and mouth, etc., ranging from tens to hundreds. Each mask has a corresponding Chinese and English name (e.g., nasolabial folds have different names). The data sampling module first counts the number of non-zero masks and sorts them. The fewer the number of non-zero masks, the easier they are to sample, which helps the model learn more refined masks. Figure 4As shown, the sampling dimension module samples masks ranging from [1-N] (N is set to 10 in this scheme), then splits them into two paths. One path uses Masks as MasksRef (reference image mask) and Image as ImageRef (reference image) to perform affine transformations to obtain MasksTarget and ImageTarget. The other path enters the FusionMask module, which merges these masks into a single mask and sends them to the sampling pool. For ease of explanation, let's assume these masks are denoted as mask_vec (vectors), there are a total of mask_num, and the merged mask is denoted as mask_fusion. The sampling pool contains six modules: sampling points, sampling boxes, fine-grained sampling masks, pseudo-masks constructed from sampling, coarse-grained sampling masks, and sampled text. During training, 1-4 modules are randomly sampled from the sampling pool each time. Among them, fine-grained sampling masks, pseudo-masks constructed from sampling, and coarse-grained sampling masks are mutually exclusive, and only one of them is selected each time. The results are then sent to the prompt encoding module. For ease of description, let's assume that a total of M mask_fusions were sampled for an image, i.e., mask_fusion_m.

[0150] 1) Sampling points: To avoid ambiguity when users provide single-point prompts, this solution first samples a positive sample point (positive point refers to the foreground point, negative point refers to the background point) for each sub-mask in mask_vec. Then, a number in the range [mask_num-N] is randomly generated. If the number is greater than mask_num, the remaining positive or negative sample points will be randomly sampled again from mask_fusion.

[0151] 2) Sampling Box: Similarly, to avoid ambiguity when providing a single box suggestion to the user, this solution first calculates the outermost contour of each sub-mask in mask_vec, and then samples a box from these contours. The sampling principle is that the more contour points there are, the easier it is to sample. Then, a number in the range [mask_num-N] is randomly generated. If this number is greater than mask_num, the remaining boxes will be randomly sampled again from the outer contour set.

[0152] 3) Fine-grained Mask Sampling: In order to enable the model to segment the user-drawn mask according to what the user draws, this solution adds a fine-grained mask strategy so that the model has the ability to reconstruct, that is, directly feed the mask_fusion into the network to learn (label closeness).

[0153] 4) Sampling coarse mask: In order to enable the model to accurately understand the doodle content and segment the corresponding mask when the user doodles, and improve the efficiency of interactive segmentation, this solution will generate some doodle content for each sub-mask in mask_vec, such as discrete points, brushstrokes, lines of different thicknesses, sketches of different shapes, etc.

[0154] 5) Pseudo-Mask Constructed by Sampling: In terms of data augmentation, this scheme constructs a pseudo-mask and samples negative sample points from it. During training, the negative sample points and the pseudo-mask are fed together into the cue encoding module, avoiding multiple interactions and gradient updates during training, thus saving training time and GPU memory overhead. The pseudo-mask is constructed by performing affine transformations, morphological processing, and edge detection on mask_fusion, ultimately randomly sampling 1-N negative sample points (N is set to 8 in this scheme).

[0155] 6) Sampling text: Each sub-mask in mask_vec has a corresponding text name, and each text name has multiple synonyms. This scheme will randomly select from these synonyms, concatenate them, and send them to the prompt encoding module.

[0156] II. Prompt Encoding Module

[0157] like Figure 7 As shown, the prompt encoding module receives four inputs from the sampling module: text, points, boxes, and masks. The text is processed by the CLIP model. CLIP (Contrastive Language-Image Pretraining) is a powerful multimodal model proposed by OpenAI that understands the relationship between images and text. This scheme uses its text encoder to extract text features. The extracted features are passed through a linear projection layer (FC+LN) to transform them into the same embedding space (dimension, feature length) as other sparse prompts (points, boxes), and then concatenated and fed into the mask decoder. The point embedding module (EmbedPoints), box embedding module (EmbedBoxes), and mask embedding module (EmbedMasks) all inherit from the SAM2 prompt encoder. During CLIP training, the parameters are frozen and not updated, while the other boxes have trainable parameters. In the text encoder, supervised training is performed on multiple similar texts. During inference, for the flexibility of text prompts (i.e., allowing users to suggest non-training text words or sentences), this scheme uses CLIP to pre-extract all training text feature vectors, and then uses similarity matching to find the feature vector closest to the user's text prompt as input.

[0158] III. Reference Diagram Network Structure Module

[0159] The reference network structure module designed in this scheme, such as... Figure 6 As shown, the overall architecture resembles a Siamese symmetric network. Solid boxes represent learnable parameters, dashed boxes represent parameter freezing, and bold solid boxes represent inputs and outputs. In the left half of the network, all parameters can be learned and updated. The right half of the network comes from an improved version of SAM2. This solution redesigns the SAM2 decoder module and applies gated attention fusion to the two image encoder modules from SAM2. Parameter freezing in image encoder module 2 and cue encoder module 2 avoids catastrophic forgetting of the model during training due to data bias from different domains. Image encoder module 1 and cue encoder module 1 are used to train on domain-specific data, avoiding excessive impact on SAM2's originally excellent feature extraction capabilities. Cue encoder module 1 receives additional text cue compared to cue encoder module 2. `Image` represents the corresponding image data. After passing through the image encoder module and the gated attention fusion module, an `ImageEmbedRef` reference image embedding vector is obtained. This embedding vector can be used as input to the subsequent target network. Detailed information about the decoder module and the gated attention module will be introduced later.

[0160] IV. Gated Attention Fusion Module

[0161] like Figure 5As shown, this is the gated attention fusion module designed for this scheme. Solid boxes represent learnable parameters, and bold solid boxes represent inputs and outputs. It aims to improve feature representation by weighted fusion of multiple features through an attention mechanism. This module receives two input features, Feats1 and Feats2, which can come from features at the same level of different network structures or features at different levels of the same network structure. First, Feats1 and Feats2 are concatenated along the channel dimension. The concatenated features are then fed into the ChannelAttnFusion module. This module performs global pooling to obtain global features along the channel dimension. These features are then processed through convolution (Conv), GELU activation function, and Sigmoid activation to generate channel-level attention weights. These attention weights are then broadcast to the spatial dimension of the original input for weighted feature fusion. The weighted features are then fed into the SpatialAttnFusion module: they undergo convolution (Conv), layer normalization (LN), GELU, convolution (Conv), and sigmoid to generate the final gate weights (Gated), which control the fusion of the two feature streams. Next, Gated * Feats1 and (1 - Gated) * Feats2 are calculated, adaptively adjusting the contributions of the two feature streams using the gate weights. Finally, an addition operation is used to fuse the two parts, yielding the final output feature. This module adaptively adjusts the information contributions of Feats1 and Feats2 to strengthen important features and suppress redundant information, thereby improving the expressive power of the fused features. The design combines channel attention and spatial attention, ensuring that the fusion process considers both global information and preserves local details, contributing to improved model discriminative ability.

[0162] V. Decoder Module

[0163] The internal processing flow of decoder module 1 in the network structure shown in the figure is as follows: Figure 8 As shown, the internal processing flow of decoder module 2 is as follows: Figure 9As shown, the two-stage Transformer of TwoWayTransformer and the multilayer perceptron of MLP both inherit the design of the original SAM2. ImageEmbed is image embedding, and both dense and sparse cues come from the output of the cue encoding module. Dense cues are transformed from mask cues, and sparse cues are transformed from dot cues, box cues, and text cues. The features after the two-stage Transformer (256x64x64) are then upsampled (enlarged) layer by layer and fused with shallow features (texture and detail features) using gated attention. The fused vector is then multiplied by the vector obtained from the multilayer perceptron to obtain the final output mask. Decoder module 1 integrates shallow features of 16x (64x64), 8x (128x128), 4x (256x256), 2x (512x512), and 1x (1024x1024), respectively. Decoder module 2 integrates shallow features of 8x (128x128), 4x (256x256), and 2x (512x512), respectively. Decoder module 1 is mainly used to handle high-precision data in the skin measurement field to ensure fine-grained segmentation. Decoder module 2 mainly plays an auxiliary role, maintaining the original accuracy of SAM2 while improving the fineness of segmentation and assisting decoder module 1 in generalizing to unknown fields.

[0164] VI. Target Graph Network Structure Module

[0165] like Figure 3As shown, this represents the network structure modules of the target graph. Solid boxes indicate learnable parameters, dashed boxes indicate frozen parameters, and bold solid boxes indicate inputs and outputs. Image encoding module 1, image encoding module 2, gated attention fusion module 1, and gated attention fusion module 2 are identical to the network structure in the reference graph and maintain their weights frozen during training on the target graph. The memory encoding module and memory attention module inherit weights from SAM2 and maintain their weights frozen during training. Decoder module 1 and decoder module 2 (learnable parameters) are also identical to the network structure in the reference graph, except that each independently replicates a two-stage Transformer and two multilayer perceptrons (MLPs). During training, except for these independent modules, the other modules maintain their weights frozen. In the sampling module, we have obtained MasksTarget and ImageTarget, MasksRef and ImageRef. ImageRef is processed through a reference graph network structure to obtain a reference graph embedding vector (ImageEmbedRef). The reference graph Mask (MasksRef) is processed through the SAM2 memory encoding module to obtain a Mask embedding vector. ImageTarget is processed through a Siamese symmetric network image encoding module and a gated attention fusion module, and then summed to obtain the target graph embedding vector. The reference graph embedding vector, target graph embedding vector, and Mask embedding vector are fed together into the memory attention module to obtain aligned embedding vectors. Finally, the aligned embedding vectors are processed through the decoder module to obtain the final mask output.

[0166] VII. Adaptive Asymmetric F1 Loss (AAF1)

[0167] This scheme is an improvement on the Cyclical Focal Loss. The original loss function is mainly used for semantic segmentation, while the loss function of this scheme can be used not only for semantic segmentation but also for SAM of interactive segmentation, especially for open-world interactive segmentation in the skin measurement domain. To make it easier to understand the loss function designed in this scheme, we assume that in N×C×M×H×W, N represents the batch, such as N images, C represents the number of masks sampled from one image, M represents the number of masks output by the reference graph network structure (or the target graph network structure), and H and W represent the corresponding width and height.

[0168] in, This indicates the input field.

[0169] Let M represent the target, where M is obtained through broadcasting. This indicates that sigmoid is activated.

[0170] First, the input is transformed using a sigmoid function, as shown in formula (1) below:

[0171] (1)

[0172] The calculation obtained in formula (1) The C and M channels of Y are first interchanged, and then flattened as shown in the following formulas (2) and (3):

[0173] (2)

[0174] (3)

[0175] The positive and negative examples of the F1-score are calculated as shown in the following formulas (4) and (5):

[0176] (4)

[0177] (5)

[0178] At this time, it was obtained , The dimensions are all NM.

[0179] The asymmetric weights are calculated as shown in formulas (6) and (7) below:

[0180] (6)

[0181] (7)

[0182] in, , , All of these are hyperparameters, and the results obtained at this time , The dimensions are all NM×CHW, and the average of the last dimension is calculated as shown in the following formula (8):

[0183] (8)

[0184] in, The dimensions are all NM, and the final AAF1 loss is shown in the following formula (9):

[0185] (9)

[0186] in, The dimension is also NM.

[0187] It is understood that in this embodiment, when X applies input to all pixels, it is called global AAF1. This means that the hyperparameters are set to: =10、 =1、 =2; when X uses PointRend (point sampling rendering), it is called local AAF1, using This indicates that the hyperparameters are set to [value]. =2、 =4、 =2. PointRend can adaptively sample the uncertain regions of the mask and apply stronger supervision to these key regions. PointRend can be used for optimization of local uncertain regions. This scheme applies both global and local AAF1 and applies a coefficient of 2 to the global AAF1 to obtain the final loss value as shown in the following formula (10):

[0188] (10)

[0189] During training, the training method used in this application only backpropagates the channel with the minimum loss among the M channels (Note: M refers to M in the loss dimension NM of Equation 9). This scheme applies both global and local losses, performing backpropagation for both the global and local minimum losses. This design effectively guides multiple output channels of the model to learn different segmentation characteristics. Total loss function The final dimension is determined by the batch size N and the number of model-defined output masks. This means that the loss function allows each sample to independently select the optimal output channel, whereas traditional methods are based on mask-level optimization, which may introduce noise, leading to more complex and unstable training. The sample-level optimization designed in this approach allows each sub-network to focus more on specific scenarios during backpropagation. For example, high-precision skin measurement data differs significantly from everyday shooting data. This loss function allows high-precision skin measurement data to focus on learning the left half of the Siamese network, while SAM data focuses on learning the right half, with both working synergistically. When blurred cues appear, this loss function can smooth out noise and implicitly filter out more general features, thereby effectively improving the stability of model training.

[0190] The beneficial effects of the technical solution in this application include:

[0191] A skin segmentation method and system based on SAM2 are proposed for the first time. This method or system has significant advantages:

[0192] (1) Rich prompt types and sampling methods: This supports Chinese and English prompts to segment various dimensions of skin measurement (pigmentation, wrinkles, pores, etc.), and supports positive prompts, negative prompts, box prompts, graffiti mask prompts, fine mask prompts, etc. Each prompt has a unique sampling method designed to improve efficiency and reduce memory usage (for example, current methods such as SAM are interactive sampling during training, which will significantly occupy memory and reduce training efficiency).

[0193] (2) Model fine-tuning system for skin measurement: a twin symmetric network and gated attention fusion module were proposed, and the SAM2 decoder was redesigned. Based on the original SAM2 interactive segmentation capability, the precision was further improved, and it can also be applied to fine-tuning professional domain data without causing catastrophic forgetting of SAM2.

[0194] (3) Co-training method of reference image and target image: A unique sampling method and supervision method for reference image and target image are designed. Given any number of reference images, the model can segment the specified target dataset based on the information of the reference images. In addition, this method also has very good results for cases with clear or ambiguous boundaries.

[0195] (4) Adaptive Asymmetric F1 Loss (AAF1): A new loss function, AAF1 loss, is proposed, which can be used to solve the problems of segmentation precision, prompt ambiguity, and sample-level optimization, ensuring that different datasets can each select the optimal output channel.

[0196] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0197] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0198] In one exemplary embodiment, such as Figure 10 As shown, an image processing apparatus is provided, including: an extraction module 1002, a determination module 1004, a processing module 1006, and a training module 1008, wherein:

[0199] The extraction module 1002 is used to extract features from the target sample image through the first image encoder and the second image encoder of the image segmentation model, respectively, to obtain the first image features and the second image features.

[0200] The determining module 1004 is used to determine the target embedding features of the target sample image based on the first image features and the second image features; wherein the target embedding features are obtained by fusing the first attention features and the second attention features, and the first attention features and the second attention features are determined based on the first image features and the second image features.

[0201] The processing module 1006 is used to perform attention processing on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image to obtain aligned embedding features.

[0202] The training module 1008 is used to train the image segmentation model based on the aligned embedding features, and to perform image processing on the image to be segmented based on the trained image segmentation model.

[0203] In one embodiment, the processing module is further configured to perform gated attention fusion processing on the first image features and the second image features to obtain the first attention features and the second attention features; perform an addition operation on the first attention features and the second attention features to obtain fused features; and use the fused features as the target embedding features of the target sample image.

[0204] In one embodiment, the apparatus further includes: a stitching module, configured to stitch the first image feature and the second image feature together to obtain stitched features; and a processing module, configured to perform channel attention fusion processing on the stitched features to obtain channel-level stitched features; and perform spatial attention fusion processing on the channel-level stitched features to obtain the first attention feature or the second attention feature.

[0205] In one embodiment, the processing module is further configured to perform global pooling on the splicing features to obtain global features at the channel level; the determining module is further configured to determine channel-level attention weights based on the global features; and determine the channel-level splicing features based on the attention weights.

[0206] In one embodiment, the processing module is further configured to perform convolution processing on the channel-level stitched features to obtain gating weights; the determining module is further configured to determine a first sub-attention feature of the first image feature based on the gating weights; determine a second sub-attention feature of the second image based on a preset value and the gating weights; the processing module is further configured to perform an addition operation on the first sub-attention feature and the second sub-attention feature to obtain the first attention feature or the second attention feature.

[0207] In one embodiment, the extraction module is further configured to extract features from the reference image using the third and fourth image encoders of the dual-path model, respectively, to obtain third image features and fourth image features; the determination module is further configured to determine the reference embedding features of the reference image based on the third and fourth image features using the dual-path model; wherein the reference embedding features are obtained by fusing the third and fourth attention features, and the third and fourth attention features are determined based on the third and fourth image features; the extraction module is further configured to extract features from the input sampling points, sampling boxes, sampling mask images, and sampling text using the first prompt encoder of the dual-path model, to obtain the third image features and fourth image features. The processing module is further configured to: 1) extract features from the input sampling points, sampling boxes, and sampling mask images using the second cue encoder of the dual-path model to obtain a second cue feature; 2) decode the third attention feature and the first cue feature using the first decoder of the dual-path model to obtain a first decoded feature; and 3) decode the fourth attention feature and the second cue feature using the second decoder of the dual-path model to obtain a second decoded feature; and 4) train the dual-path model based on the first decoded feature and the second decoded feature, and process other reference images based on the trained dual-path model to obtain reference embedding features of the other reference images.

[0208] In one embodiment, the apparatus further includes: an acquisition module, configured to acquire reference embedding features of the reference image obtained by processing the reference image through the dual-path model; a processing module, configured to decode the aligned embedding features through each decoder in the image segmentation model to obtain a first decoded feature and a second decoded feature; and a training module, configured to train the image segmentation model based on the first decoded feature and the second decoded feature, and perform image processing on the image to be segmented based on the trained image segmentation model.

[0209] In one embodiment, the determining module is further configured to determine local output probability values ​​and global output probability values ​​based on the target sample image, the first decoding feature, and the second decoding feature, respectively, until the loss value determined based on the local output probability value and the global output probability value satisfies a preset loss condition, and then stop training to obtain the trained image segmentation model.

[0210] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0211] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0212] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0213] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0214] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0215] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0216] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0217] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0218] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0219] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: The third and fourth image encoders of the dual-path model are used to extract features from the reference image to obtain the third image features and the fourth image features. Using the dual-path model, reference embedding features of the reference image are determined based on the third image features and the fourth image features; wherein, the reference embedding features are obtained by fusing the third attention features and the fourth attention features, and the third attention features and the fourth attention features are determined based on the third image features and the fourth image features; The first image encoder and the second image encoder of the image segmentation model are used to extract features from the target sample image to obtain the first image features and the second image features. The target embedding features of the target sample image are determined based on the first image features and the second image features; wherein the target embedding features are obtained by fusing the first attention features and the second attention features, and the first attention features and the second attention features are determined based on the first image features and the second image features; Attention processing is performed on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image to obtain the aligned embedding feature; The image segmentation model is trained based on the aligned embedding features, and image processing is performed on the image to be segmented based on the trained image segmentation model.

2. The method according to claim 1, characterized in that, Determining the target embedding features of the target sample image based on the first image features and the second image features includes: Gated attention fusion processing is performed on the first image features and the second image features to obtain the first attention features and the second attention features; The first attention feature and the second attention feature are added together to obtain the fused feature. The fused features are used as the target embedding features of the target sample image.

3. The method according to claim 2, characterized in that, The step of performing gated attention fusion processing on the first image features and the second image features to obtain the first attention features and the second attention features includes: The first image features and the second image features are concatenated to obtain the concatenated features; The splicing features are subjected to channel attention fusion processing to obtain channel-level splicing features; Spatial attention fusion processing is performed on the channel-level splicing features to obtain the first attention feature or the second attention feature.

4. The method according to claim 3, characterized in that, The process of performing channel attention fusion on the splicing features to obtain channel-level splicing features includes: The spliced ​​features are subjected to global pooling to obtain global features in the channel dimension; Channel-level attention weights are determined based on the global features; The channel-level splicing features are determined based on the attention weights.

5. The method according to claim 3, characterized in that, The step of performing spatial attention fusion processing on the channel-level splicing features to obtain the first attention feature or the second attention feature includes: The channel-level splicing features are convolutionally processed to obtain gating weights; The first sub-attention feature of the first image feature is determined based on the gating weight; The second sub-attention feature of the second image is determined based on the preset value and the gating weight; The first sub-attention feature and the second sub-attention feature are added together to obtain the first attention feature or the second attention feature.

6. The method according to claim 1, characterized in that, Before the first image encoder and the second image encoder of the image segmentation model extract features from the target sample image to obtain the first image features and the second image features, the method further includes: The first prompt encoder of the dual-path model extracts features from the input sampling points, sampling boxes, sampling mask images, and sampling text to obtain first prompt features; and the second prompt encoder of the dual-path model extracts features from the input sampling points, sampling boxes, and sampling mask images to obtain second prompt features. The first decoder of the dual-path model decodes the third attention feature and the first cue feature to obtain the first decoded feature; and the second decoder of the dual-path model decodes the fourth attention feature and the second cue feature to obtain the second decoded feature. The dual-path model is trained based on the first decoding feature and the second decoding feature, and other reference images are processed based on the trained dual-path model to obtain the reference embedding features of the other reference images.

7. The method according to claim 6, characterized in that, Before performing attention processing on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image to obtain the aligned embedding feature, the method further includes: Obtain the reference embedding features of the reference image obtained by processing the reference image through the dual-path model; After performing attention processing on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image to obtain the aligned embedding features, the method further includes: The aligned embedded features are decoded by each decoder in the image segmentation model to obtain the first decoded feature and the second decoded feature. The step of training the image segmentation model based on the aligned embedding features, and then performing image processing on the image to be segmented based on the trained image segmentation model, includes: The image segmentation model is trained based on the first decoding feature and the second decoding feature, and image processing is performed on the image to be segmented based on the trained image segmentation model.

8. The method according to claim 7, characterized in that, The step of training the image segmentation model based on the first decoding feature and the second decoding feature includes: Based on the target sample image, the first decoding feature, and the second decoding feature, local output probability values ​​and global output probability values ​​are determined respectively. Training stops when the loss value determined based on the local output probability values ​​and the global output probability values ​​meets a preset loss condition, so as to obtain the trained image segmentation model.

9. An image processing apparatus, characterized in that, The device includes: An extraction module is used to extract features from a reference image using a third image encoder and a fourth image encoder of a dual-path model, respectively, to obtain third image features and fourth image features. A determination module is used to determine a reference embedding feature of the reference image based on the third image feature and the fourth image feature using the dual-path model; wherein the reference embedding feature is obtained by fusing a third attention feature and a fourth attention feature, and the third attention feature and the fourth attention feature are determined based on the third image feature and the fourth image feature; the extraction module is also used to extract features from a target sample image using a first image encoder and a second image encoder of an image segmentation model, respectively, to obtain first image features and second image features. The determining module is further configured to determine the target embedding features of the target sample image based on the first image features and the second image features; wherein the target embedding features are obtained by fusing the first attention features and the second attention features, and the first attention features and the second attention features are determined based on the first image features and the second image features; The processing module is used to perform attention processing on the target embedding feature, the reference embedding feature of the reference image, and the mask embedding feature of the mask image corresponding to the reference image to obtain aligned embedding features; The training module is used to train the image segmentation model based on the aligned embedding features, and to perform image processing on the image to be segmented based on the trained image segmentation model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images