Image processing method and apparatus

Abstract

The application provides an image processing method and device, the method comprising: performing feature extraction on a user-input original image to obtain a semantic feature; determining, by using an encoder, a plurality of semantic feature components and a plurality of residual feature components aligned with a plurality of semantic dimensions according to the semantic feature and a parameter matrix, the parameter matrix being obtained by training the encoder using a training sample set, elements on a diagonal line in the parameter matrix being constrained to be positive and elements above the diagonal line being constrained to be zero, the plurality of semantic dimensions being extracted from the training sample set, the semantic feature components including semantic information of the original image in corresponding semantic dimensions, and the residual feature components including semantic information of the original image outside the plurality of semantic dimensions; and determining a target image according to the semantic feature, the plurality of semantic feature components, and the plurality of residual feature components.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method and apparatus. Background Technology

[0002] Decoupled representation learning aims to decompose observed data into semantically interpretable and independent latent factors, a learning paradigm inspired by how biological intelligence perceives and understands the world. In this form of representation, different representational attributes (such as facial expressions, wrinkles, and accessories) exhibit specific sensitivities only to changes in specific factors. Learning such decoupled representations has significant value in multiple fields, including computer vision, generative modeling, and interpretable artificial intelligence. Most existing decoupled representation learning methods are based on the Beta variational autoencoder framework, whose core idea is to promote the orthogonality of representations through weighted KL divergence (relative entropy) penalties, thereby achieving better decoupling. Building upon this, -TCVAE (a variant of the Beta variational autoencoder framework), DIP-VAE (a variant of the Beta variational autoencoder framework), FactorVAE (a variant of the Beta variational autoencoder framework), RF-VAE (a variant of the Beta variational autoencoder framework) and Variants such as TCVAE (a variant of the Beta variational autoencoder framework) have driven the development of this paradigm through regularization optimizations. However, based on -VAE models incorrectly equate decoupling with strict statistical orthogonality, thus failing to learn intrinsic semantics. Specifically, existing methods suffer from two long-standing and fundamental problems: First, they oversimplify semantic decoupling to statistical orthogonality. Existing methods typically reduce "decoupling" to a statistical compression task, achieving it through strict orthogonality constraints and conflating statistical patterns with semantic factors. However, this process ignores the non-orthogonal nature of real-world concepts, causing the model to be unable to perceive meaningful semantics. Second, there is an inherent trade-off between decoupling performance and reconstruction fidelity. Forcing orthogonality usually enhances decoupling capabilities at the expense of reconstruction fidelity. This stems from the over-compression of information bottlenecks caused by rigid orthogonality penalties, which prioritize the independence of factors while filtering out the fine-grained information needed for reconstruction. Therefore, there is an urgent need for a scheme that can reconstruct images without relying on strict orthogonality assumptions. Summary of the Invention

[0003] This application provides an image processing method and apparatus.

[0004] One embodiment of this application provides an image processing method, the method comprising: Semantic features are obtained by extracting features from the original image input by the user. An encoder is used to determine multiple semantic feature components and multiple residual feature components aligned with multiple semantic dimensions based on the semantic features and parameter matrix. The parameter matrix is ​​obtained by training the encoder using a training sample set. In the parameter matrix, all elements on the diagonal are constrained to be positive and all elements above the diagonal are constrained to be zero. The multiple semantic dimensions are extracted from the training sample set. The semantic feature components include the semantic information of the original image in the corresponding semantic dimension, and the residual feature components include the semantic information of the original image outside the multiple semantic dimensions. The target image is determined based on the semantic features, the plurality of semantic feature components, and the plurality of residual feature components.

[0005] The step of determining the target image based on the semantic features, the plurality of semantic feature components, and the plurality of residual feature components includes: The semantic features are forward diffused to obtain noisy semantic features; The noisy semantic features are back-diffused based on the multiple semantic feature components and the multiple residual feature components to obtain the target image.

[0006] The method further includes: The training sample set is semantically extracted using a language model to obtain multiple initial semantic dimensions and multiple text descriptions corresponding to the initial semantic dimensions. The language model is used to generate multiple semantically identical prompt texts corresponding to each text description in the initial semantic dimension; The total coverage of the initial semantic dimension in the training sample set is determined based on multiple text descriptions corresponding to the initial semantic dimension. Determine the semantic similarity between each text description corresponding to the initial semantic dimension and each verification sample image in the verification sample set; The mean semantic similarity difference of the initial semantic dimension is determined based on the highest and second-highest semantic similarity between each verification sample image and each text description; Multiple initial semantic dimensions that meet the target conditions are determined as the multiple semantic dimensions. The target conditions are that the total coverage of the initial semantic dimensions is within a preset coverage range, the initial semantic dimensions have at least two text descriptions with coverage greater than the preset coverage, and the average semantic similarity difference of the initial semantic dimensions is greater than a preset difference.

[0007] The method further includes: Feature extraction is performed on each sample image in the training sample set to obtain the corresponding sample semantic features; The initial encoder is trained based on the sample semantic features, the initial parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions to obtain the candidate encoder and the candidate parameter matrix. Based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions, the initial target dimension predictor and the initial complementary dimension predictor corresponding to each semantic dimension are trained to obtain the target dimension predictor and the complementary dimension predictor corresponding to each semantic dimension. The candidate encoder is trained based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, multiple prompt texts corresponding to the text descriptions, and the target dimension predictor and complementary dimension predictor corresponding to each semantic dimension, to obtain the encoder and the parameter matrix.

[0008] The step of training the initial encoder based on the sample semantic features, the initial parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions includes: The initial encoder determines multiple first sample semantic feature components aligned with multiple semantic dimensions based on the sample semantic features and the initial parameter matrix. The sample semantic feature components include the semantic information of the sample image in the corresponding semantic dimension. The first mask and the first pseudo-label corresponding to each semantic dimension of the sample semantic features are determined based on multiple first sample semantic feature components of the sample semantic features, multiple text descriptions corresponding to the semantic dimensions, and multiple prompt texts corresponding to the text descriptions. The first loss value is determined based on the multiple first sample semantic feature components of the sample semantic features and the first mask and first pseudo label corresponding to each semantic dimension; The second loss value is determined based on multiple first sample semantic feature components of the sample semantic features, the positive sample set and the negative sample set corresponding to each semantic dimension. The positive sample set includes multiple positive sample semantic feature components corresponding to the semantic dimension, and the negative sample set includes multiple negative sample semantic feature components corresponding to the semantic dimension. A third loss value is determined based on the sample semantic features and multiple first sample semantic feature components corresponding to the sample semantic features; The parameters of the initial encoder are optimized based on the first loss value, the second loss value, and the third loss value.

[0009] The step of training the initial target dimension predictor and the initial complementary dimension predictor for each semantic dimension based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions includes: The candidate encoder uses the sample semantic features and the candidate parameter matrix to determine multiple second sample semantic feature components aligned with multiple semantic dimensions; The second mask and the second pseudo-label corresponding to each semantic dimension of the sample semantic feature are determined based on the multiple second sample semantic feature components corresponding to the sample semantic feature, the multiple text descriptions corresponding to the semantic dimension, and the multiple prompt texts corresponding to the text descriptions. The initial target dimension predictor predicts the second sample semantic feature components corresponding to each semantic dimension in the sample semantic features according to the initial target dimension predictor, and obtains the first predicted value of each semantic dimension of the second sample semantic features. The input of the initial target dimension predictor is the second sample semantic feature component corresponding to the semantic dimension. The second sample semantic feature components corresponding to each semantic dimension in the sample semantic features are predicted according to the initial complementary dimension predictor corresponding to each semantic dimension, so as to obtain the second predicted value of each semantic dimension of the second sample semantic features. The input of the initial complementary dimension predictor is the second sample semantic feature components of all semantic dimensions except the corresponding semantic dimension. The fourth loss value is determined based on the first predicted value, the second predicted value, the second mask, and the second pseudo-label of each semantic dimension of the sample semantic features; The parameters of the initial target dimension predictor and the initial complementary dimension predictor corresponding to each semantic dimension are optimized based on the fourth loss value.

[0010] The step of training the candidate encoder based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, multiple prompt texts corresponding to the text descriptions, and the target dimension predictor and complementary dimension predictor corresponding to each semantic dimension includes: Based on the target dimension predictor corresponding to each semantic dimension, the second sample semantic feature components corresponding to each semantic dimension in the second sample semantic feature are predicted to obtain the third predicted value of each semantic dimension of the second sample semantic feature. Based on the complementary dimension predictor corresponding to each semantic dimension, the second sample semantic feature components corresponding to each semantic dimension in the sample semantic features are predicted to obtain the fourth predicted value of each semantic dimension of the second sample semantic features. The fifth loss value is determined based on the third predicted value, the fourth predicted value, the second mask, and the second pseudo-label of each semantic dimension of the sample semantic features; The first loss value is re-determined based on the multiple second sample semantic feature components of the sample semantic features and the second mask and second pseudo-label corresponding to each semantic dimension; The second loss value is re-determined based on multiple second sample semantic feature components, the positive sample set, and the negative sample set of the sample semantic features; The third loss value is re-determined based on the semantic features of the sample; The parameters of the candidate encoder are optimized based on the first loss value, the second loss value, the third loss value, and the fifth loss value.

[0011] The process of determining the mask and pseudo-label corresponding to each semantic dimension of the sample semantic features includes: Based on multiple text descriptions corresponding to the semantic dimension and multiple prompt texts corresponding to the text descriptions, the semantic features of each feature category of the semantic dimension are determined. The feature categories include a first feature category and a second feature category that are mutually exclusive, and a third feature category that does not belong to either the first feature category or the second feature category. Based on the similarity between each sample semantic feature component of the sample semantic feature and the category semantic features of each feature category corresponding to the semantic dimension of the sample semantic feature component; The feature category with the highest similarity is determined as the feature category to which the semantic feature component of the sample belongs; The mask and pseudo-label of the semantic dimension corresponding to the semantic feature of the sample are determined based on the feature category to which the sample semantic feature component belongs.

[0012] The method further includes: The variance of the semantic dimension is determined based on the semantic feature components of the samples in the training sample set whose feature categories are the first feature category and the second feature category. If the variance is less than a preset variance, then the frequency of occurrence of the semantic dimension in all positive sample sets in the training sample set is determined; The correction weight of the semantic dimension is determined based on the occurrence frequency. The correction weight is used to correct the sample semantic feature components belonging to the semantic dimension when determining the first loss value and the second loss value.

[0013] Another embodiment of this application provides an image processing apparatus, the apparatus comprising: The processing module is used to extract features from the original image input by the user to obtain semantic features; The calculation module is used to determine multiple semantic feature components and multiple residual feature components aligned with multiple semantic dimensions using the encoder based on the semantic features and parameter matrix. The parameter matrix is ​​obtained by training the encoder using a training sample set. In the parameter matrix, all elements on the diagonal are constrained to be positive and all elements above the diagonal are constrained to be zero. The multiple semantic dimensions are extracted from the training sample set. The semantic feature components include the semantic information of the original image in the corresponding semantic dimension, and the residual feature components include the semantic information of the original image outside the multiple semantic dimensions. The generation module is used to determine the target image based on the semantic features, the plurality of semantic feature components, and the plurality of residual feature components.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0015] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, wherein: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0016] Figure 1 A flowchart of an image processing method according to an embodiment of this application is shown; Figure 2 A flowchart of an image processing method according to another embodiment of this application is shown; Figure 3 A flowchart of an image processing method according to another embodiment of this application is shown; Figure 4 A flowchart of an image processing method according to another embodiment of this application is shown; Figure 5 A flowchart of an image processing method according to another embodiment of this application is shown; Figure 6 A flowchart of an image processing method according to another embodiment of this application is shown; Figure 7 A flowchart of an image processing method according to another embodiment of this application is shown; Figure 8 A flowchart of an image processing method according to another embodiment of this application is shown; Figure 9A flowchart of an image processing method according to another embodiment of this application is shown; Figure 10 A schematic diagram of the structure of an image processing apparatus according to an embodiment of this application is shown; Figure 11 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0017] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] To reconstruct images without relying on strict orthogonality assumptions, capture meaningful semantic information in the image to be reconstructed, and thereby simultaneously improve decoupling performance and reconstruction fidelity, an embodiment of this application provides an image processing method, such as... Figure 1 As shown, the method includes: Step 101: Extract features from the original image input by the user to obtain semantic features.

[0019] In this embodiment, an image processing model is used to reconstruct the original image. The image processing model includes a Clip model (Contrastive Language-Image Pre-training), an encoder, and a diffusion model.

[0020] The Clip model is used to extract the underlying latent variables of the user-input raw image, which are then normalized using the L2 norm (Euclidean norm). The basic latent variables Input encoder, encoder output feature mean and characteristic covariance parameters .

[0021] The original image is reparameterized using the following formula:

[0022] in, These are semantic latent variables, i.e., semantic features. The noise is standard Gaussian random noise, following a standard multivariate normal distribution with a mean of zero and covariances forming an identity matrix. The characteristic mean , Characteristic covariance parameter .

[0023] Step 102: The encoder determines multiple semantic feature components and multiple residual feature components aligned with multiple semantic dimensions based on the semantic features and parameter matrix. The parameter matrix is ​​obtained by training the encoder using a training sample set. In the parameter matrix, all diagonal elements are constrained to be positive and all elements above the diagonal are constrained to be zero. The multiple semantic dimensions are extracted from the training sample set. The semantic feature components include the semantic information of the original image in the corresponding semantic dimension, and the residual feature components include the semantic information of the original image outside the multiple semantic dimensions.

[0024] To break the latent space rotational symmetry, it is necessary to utilize the trained encoder based on the parameter matrix. A linear transformation is performed on the semantic features to align the semantic information contained in the semantic feature components with the corresponding semantic dimensions, thereby obtaining multiple semantic feature components aligned with multiple semantic dimensions.

[0025] The parameter matrix is ​​obtained by optimizing the internal parameters (including the parameter matrix) of the encoder during training. The parameter matrix is ​​an invertible lower triangular matrix, in which all elements on the diagonal are constrained to be positive, and all elements above the diagonal are constrained to be zero.

[0026] For example, as shown in Table 1, Table 1 presents a parameter matrix, where the diagonal elements of the parameter matrix are... , , All elements are positive, and the element at the top right of the diagonal is set to 0.

[0027] Table 1

[0028] Multiple semantic dimensions are extracted from the training sample set and can represent various features in the training sample set. For example, semantic dimensions include color, material, shape, pose, lighting, and background.

[0029] The semantic feature component, aligned with the semantic dimension, includes the semantic information of the original image at the corresponding semantic dimension. For example, the semantic feature component corresponding to color in the semantic dimension includes the semantic information that the hat of the person in the original image is red.

[0030] In addition, since the original image includes not only semantic information corresponding to multiple semantic dimensions, but also semantic information outside of these multiple semantic dimensions, this part of semantic information is necessary for high reconstruction fidelity. Therefore, when extracting semantic features and semantic feature components, the total number of dimensions needs to be greater than the total number of semantic dimensions. The excess part is the residual feature component. The residual feature component is used to carry the semantic information outside of these multiple semantic dimensions in the original image, thereby improving the reconstruction fidelity when determining the target image.

[0031] Step 103: Determine the target image based on the semantic features, the plurality of semantic feature components, and the plurality of residual feature components.

[0032] The diffusion model is used to generate target images based on semantic features and residual features as constraints.

[0033] Because of the rotational symmetry of semantic features, this feature can cause semantic supervision to be bypassed during training, resulting in multiple semantic feature components output by the encoder after training not being able to align with the corresponding multiple semantic dimensions.

[0034] In the aforementioned scheme, a linear transformation of semantic features is performed using a parameter matrix whose diagonal elements are all constrained to be positive and whose elements above the diagonal are all constrained to be zero. This completely eliminates the zero-cost orthogonal degrees of freedom, breaking the rotational symmetry of semantic features. This allows multiple semantic feature components output by the encoder to align with multiple semantic dimensions, preventing them from being influenced by unguided semantic dimensions (i.e., semantic dimensions other than those extracted from the training dataset), which could affect image reconstruction. A stable correspondence between semantic dimensions and semantic feature components is established without forcing global orthogonality. This achieves image reconstruction without relying on strict orthogonality assumptions, capturing meaningful semantic information in the image to be reconstructed, and simultaneously improving both decoupling performance and reconstruction fidelity.

[0035] An image processing method is also provided in one example of this application, such as Figure 2 As shown, determining the target image based on the semantic features, the plurality of semantic feature components, and the plurality of residual feature components includes: Step 201: Perform forward diffusion on the semantic features to obtain noisy semantic features.

[0036] A diffusion model is used to forward diffuse semantic features, which involves repeatedly adding random noise to the semantic features until a preset condition is met, resulting in noisy semantic features. The target condition can be either that the number of times random noise is added reaches a preset number, or that the noise intensity of the semantic features is greater than or equal to a preset threshold.

[0037] Step 202: Perform reverse diffusion on the noisy semantic features based on the multiple semantic feature components and the multiple residual feature components to obtain the target image.

[0038] Multiple semantic feature components and multiple residual feature components are used to constrain the direction of backdiffusion of noisy semantic features. For example, multiple semantic feature components and multiple residual feature components are used as conditional embeddings of U-Net (a convolutional neural network) in the diffusion model to guide the denoising process of backdiffusion. Multiple semantic feature components, multiple residual feature components, and noisy semantic features are input into U-Net. The diffusion model uses these multiple semantic feature components and multiple residual feature components to constrain the noisy semantic features, denoising them. After denoising, the images are decoded to finally generate the target image.

[0039] In the above scheme, noisy semantic features are obtained by repeatedly superimposing random noise onto semantic features until a preset condition is met. Then, multiple semantic feature components and multiple residual feature components, which are aligned one-to-one with multiple semantic dimensions, are used to constrain the direction of the back diffusion of the noisy semantic features. These multiple semantic feature components and multiple residual feature components are used as conditional embedding information of U-Net, and the denoising process of the noisy semantic features is completed based on the constraints formed by these multiple semantic feature components and multiple residual feature components. After denoising, decoding is performed to generate the target image. By constraining the direction of the back diffusion denoising through multiple semantic feature components and multiple residual feature components aligned one-to-one with multiple semantic dimensions, the semantic deviation of the generated result can be continuously standardized, reducing the possibility of semantic deviation. This ensures that the final decoded target image conforms to the semantic information of multiple semantic dimensions and other semantic dimensions outside of these multiple semantic dimensions in the original image, thereby improving the reconstruction fidelity.

[0040] An image processing method is also provided in one example of this application, such as Figure 3 As shown, the method further includes: Step 301: Use a language model to extract semantics from the training sample set to obtain multiple initial semantic dimensions and multiple text descriptions corresponding to the initial semantic dimensions.

[0041] A multimodal large language model is used to identify each sample image in the training sample set, extracting and generating attribute-centric structured descriptions, namely, an initial semantic dimension and its corresponding multiple text descriptions, covering visual attributes such as color, material, texture, shape, pose, lighting, and background. Compared with free text, structured fields can force different images to use consistent key names, thereby significantly reducing noise caused by synonyms and avoiding ambiguity in the subsequent definition of semantic dimensions.

[0042] For example, both "blonde" and "blonde" can be mapped to the same semantic dimension "hair color", where "hair color" is the initial semantic dimension and "blonde" and "blonde" are the text descriptions corresponding to this initial semantic dimension.

[0043] After obtaining multiple text descriptions corresponding to each of the multiple initial semantic dimensions, various normalization operations can be performed on them: The first approach proposes initial semantic dimensions with clear semantic boundaries, where each field expresses only one concept. This involves merging initial semantic dimensions that are semantically similar or identical. For example, if the extracted initial semantic dimensions include "red" and "blue," both belonging to color semantics, these two initial semantic dimensions can be merged into a single "color" semantic dimension.

[0044] The second approach is to normalize field names from different images into a unified, standardized identifier. For example, if the same text descriptions extracted from multiple images are inconsistent, labeled as "duckbill cap," "baseball cap," and "fisherman's cap," then all these different text descriptions can be normalized into multiple text descriptions within the semantic dimension of "hat."

[0045] The third approach is to merge semantically equivalent but differently expressed values ​​into a small number of discrete states. For example, if multiple images extract expressions such as "wearing a hat," "having a hat on one's head," and "wearing a hat," and these three are semantically equivalent, they can be merged into a single text description, "wearing a hat."

[0046] Step 302: Use the language model to generate multiple semantically identical prompt texts corresponding to each text description in the initial semantic dimension.

[0047] A multimodal large language model is used to generate multiple semantically identical prompt texts corresponding to each text description in each initial semantic dimension.

[0048] For example, for the text description "baseball cap" in the initial semantic dimension "hat", multiple semantically identical prompt texts are generated: "The person in the picture is wearing a baseball cap", "The person in the picture is wearing a baseball cap", and "The person's head has the characteristic of wearing a baseball cap". For the text description "baseball cap" in the initial semantic dimension "hat", multiple semantically identical prompt texts are generated: "The person in the photo is wearing a baseball cap", "The image contains a person wearing a baseball cap", and "A person wearing a baseball cap appears in the picture". For the text description "fisherman's hat" in the initial semantic dimension "hat", multiple semantically identical prompt texts are generated: "The person is wearing a fisherman's hat", and "The main person in the picture is wearing a fisherman's hat".

[0049] Step 303: Determine the total coverage of the initial semantic dimension in the training sample set based on the multiple text descriptions corresponding to the initial semantic dimension.

[0050] Specifically, the number can be determined using the following formula. Coverage of text descriptions in the initial semantic dimensions in the training sample set :

[0051] in, The total number of sample images in the training sample set, if the... The sample image exists in the first... The true labels in the initial semantic dimension, then The first option can be used Sample images In the The true label on the initial semantic dimension, if the first... The sample image does not exist in the first... The true labels in the initial semantic dimension, then For the first Sample images In the The discrete values ​​at the initial semantic dimension are determined by the Clip model from the... Sample images The value obtained from this extraction can be the discrete value with the highest confidence level among the extracted results. For the first The discrete values ​​of the initial semantic dimension, i.e. the first... A textual description of an initial semantic dimension.

[0052] After determining the coverage of each text description in the initial semantic dimension in the training sample set, the coverage is summed to obtain the total coverage of the initial semantic dimension.

[0053] Step 304: Determine the semantic similarity between each text description corresponding to the initial semantic dimension and each verification sample image in the verification sample set.

[0054] The text description can be determined using the following formula. Compared with the first in the validation sample set One verification sample image semantic similarity between :

[0055] in, The first extracted by the Clip model One verification sample image Image embedding, Text description extracted by the Clip model Any corresponding prompt text Text embedding.

[0056] Step 305: Determine the mean semantic similarity difference of the initial semantic dimension based on the highest and second-highest semantic similarity between each verification sample image and each text description.

[0057] Specifically, the mean semantic similarity difference of the initial semantic dimension can be determined using the following formula. :

[0058]

[0059] in, For the initial semantic dimension and the first One verification sample image The text description with the highest semantic similarity, For the initial semantic dimension and the first One verification sample image The text description with the second highest semantic similarity. To validate the sample set, The total number of validation sample images in the validation sample set.

[0060] Step 306: Determine the multiple initial semantic dimensions that meet the target conditions as the multiple semantic dimensions. The target conditions are that the total coverage of the initial semantic dimensions is within a preset coverage range, the initial semantic dimensions have at least two text descriptions with coverage greater than the preset coverage, and the average semantic similarity difference of the initial semantic dimensions is greater than a preset difference.

[0061] The target conditions include the following three types: The first approach is to ensure that the total coverage of the initial semantic dimensions falls within a preset coverage range. In this embodiment, the preset coverage range is set to 5%-95%, meaning that initial semantic dimensions with a total coverage within the range of 5%-95% are retained. In other embodiments, the preset coverage range can be set based on specific requirements.

[0062] The second approach involves having at least two text descriptions in the initial semantic dimension with coverage rates greater than a preset coverage rate. In this embodiment, the preset coverage rate can be set to any value between 0.01 and 0.02. In other embodiments, the preset coverage rate can be set based on specific requirements.

[0063] The third approach is to set the average semantic similarity difference of the initial semantic dimension to be greater than a preset difference. In this embodiment, the preset difference can be set to any value between 0.05 and 0.1. In other embodiments, the preset difference can be set based on specific requirements.

[0064] Remove the initial semantic dimensions that do not meet any of the target conditions, retain the initial semantic dimensions that meet all the target conditions, and determine them as the semantic dimensions.

[0065] In the above scheme, firstly, a multimodal large language model is used to extract semantics from the training sample set, resulting in multiple initial semantic dimensions and multiple text descriptions corresponding to these initial semantic dimensions. This structured description reduces noise from synonyms and minimizes ambiguity in semantic dimension definitions. Then, the multimodal large language model generates multiple semantically consistent prompt texts for each text description under each semantic dimension. Next, three types of normalization are applied to the initial semantic dimensions to standardize the expression forms of the initial semantic dimensions and their corresponding text descriptions. Then, the total coverage of each initial semantic dimension in the training sample set is determined, along with the mean semantic similarity difference of the initial semantic dimensions. The initial semantic dimensions are filtered based on whether the total coverage falls within a preset coverage range to determine if the coverage of the initial semantic dimensions in the training sample set is appropriate. The presence of at least two text descriptions with coverage greater than the preset coverage rate indicates whether the initial semantic dimensions possess non-degeneracy, meaning there are at least two text descriptions in the training sample set with multiple valid data points (i.e., coverage greater than the preset coverage rate). By checking whether the mean semantic similarity difference is greater than a preset difference, we can determine whether the initial semantic dimension possesses visual separability. Specifically, this means checking if the interval between the highest and second-highest semantic similarity scores of the initial semantic dimension and each validation sample image in the validation sample set is sufficiently large (i.e., the mean semantic similarity difference is greater than the preset difference), making the semantic cues visually easily distinguishable. Using these three objective conditions to filter the initial semantic dimensions, those that do not meet any objective condition are removed, and only those that meet all conditions are retained. This effectively filters out initial semantic dimensions in the training sample set that have no practical distinguishing value, retaining semantic dimensions with stable distinguishability, reasonable sample coverage, non-degeneracy, and visual separability. It standardizes various semantically related descriptions and expands the number of semantically related cues, providing standardized, rich, and low-noise semantic foundation information for subsequent image processing. Simultaneously, it significantly reduces reliance on manual annotation and improves the cross-domain transferability of the model.

[0066] An image processing method is also provided in one example of this application, such as Figure 4 As shown, the method further includes: Step 401: Extract features from each sample image in the training sample set to obtain the corresponding sample semantic features.

[0067] Step 402: Train the initial encoder based on the sample semantic features, the initial parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions to obtain the candidate encoder and the candidate parameter matrix.

[0068] The initial encoder is trained based on the semantic features of each sample, the initial parameter matrix in the initial encoder, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to each text description. Training is completed when the preset number of training rounds or the loss function converges, resulting in a candidate encoder and a candidate parameter matrix in the candidate encoder. This enables the trained candidate encoder to learn the ability to generate target images based on sample images, i.e., the ability to reconstruct images.

[0069] Step 403: Train the initial target dimension predictor and the initial complementary dimension predictor for each semantic dimension based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions, to obtain the target dimension predictor and the complementary dimension predictor for each semantic dimension.

[0070] Based on the semantic features of each sample, the candidate parameter matrix in the candidate encoder, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to each text description, the initial target dimension predictor and the initial complementary dimension predictor for each semantic dimension are trained. Similarly, when the preset number of training rounds or the loss function converges, the training is completed, and the target dimension predictor and the complementary dimension predictor for each semantic dimension are obtained, so that the trained target dimension predictor and complementary dimension predictor can learn the ability to predict the semantic information of their corresponding semantic dimensions.

[0071] For each semantic dimension, a corresponding target dimension predictor and complementary dimension predictor are created. The target dimension predictor only takes into account the component of the corresponding semantic dimension in the sample semantic features, while the complementary dimension predictor takes into account the component of the corresponding semantic dimension in the sample semantic features, as well as the component of the semantic dimension that is complementary to the semantic dimension in the sample semantic features (i.e., all other semantic dimensions besides the semantic dimension).

[0072] Step 404: Train the candidate encoder based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, multiple prompt texts corresponding to the text descriptions, and the target dimension predictor and complementary dimension predictor corresponding to each semantic dimension to obtain the encoder and the parameter matrix.

[0073] The candidate encoder is trained based on the semantic features of each sample, the candidate parameter matrix in the candidate encoder, multiple text descriptions corresponding to each semantic dimension, multiple prompt texts corresponding to each text description, and the target dimension predictor and complementary dimension predictor corresponding to each semantic dimension. Similarly, the training is completed when the preset number of training rounds or the loss function converges, and the encoder and parameter matrix are obtained. During the training process, the image reconstruction process of the candidate encoder is supervised and constrained by the prediction ability of the target dimension predictor and complementary dimension predictor on the semantic information of the corresponding semantic dimension, so that the candidate encoder can further optimize its ability to reconstruct images under supervision and constraints.

[0074] An image processing method is also provided in one example of this application, such as Figure 5 As shown, training the initial encoder based on the sample semantic features, the initial parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions includes: Step 501: Using the initial encoder, determine multiple first sample semantic feature components aligned with multiple semantic dimensions based on the sample semantic features and the initial parameter matrix. The sample semantic feature components include the semantic information of the sample image in the corresponding semantic dimension.

[0075] The initial encoder performs a linear transformation on the semantic features of the samples based on the initial parameter matrix, thereby obtaining multiple first sample semantic feature components that are aligned one-to-one with multiple semantic dimensions.

[0076] Specifically, a linear transformation can be performed using the following formula:

[0077] in, The sample semantic features are obtained after linear transformation of the initial parameter matrix, and include multiple first sample semantic feature components that are aligned one-to-one with multiple semantic dimensions. For the parameter matrix, These are the semantic features of the samples before linear transformation.

[0078] Step 502: Determine the first mask and first pseudo-label corresponding to each semantic dimension of the sample semantic features based on the multiple first sample semantic feature components of the sample semantic features, the multiple text descriptions corresponding to the semantic dimensions, and the multiple prompt texts corresponding to the text descriptions.

[0079] Based on the multiple text descriptions corresponding to the semantic dimension and the multiple prompt texts corresponding to each text description, the first mask and the first pseudo-label of the first sample semantic feature component belonging to the semantic dimension can be determined. The pseudo-label represents whether the first sample semantic feature component of the semantic dimension conforms to the semantic information of the semantic dimension, while the mask represents the validity of the first sample semantic feature component of the semantic dimension.

[0080] Step 503: Determine the first loss value based on the multiple first sample semantic feature components of the sample semantic features and the first mask and first pseudo-label corresponding to each semantic dimension.

[0081] Specifically, the first loss value can be determined based on the following formula. :

[0082] in, For the first The semantic features of the sample image of the nth sample The first mask corresponding to the first sample semantic feature component of each semantic dimension. For the first The semantic features of the sample image of the nth sample The first sample semantic feature component of each semantic dimension For the first The semantic features of the sample image of the nth sample The first pseudo-label corresponds to the first sample semantic feature component of each semantic dimension.

[0083] Step 504: Determine a second loss value based on multiple first sample semantic feature components of the sample semantic features, the positive sample set and the negative sample set of each semantic dimension, wherein the positive sample set includes multiple positive sample semantic feature components of the corresponding semantic dimension, and the negative sample set includes multiple negative sample semantic feature components of the corresponding semantic dimension.

[0084] Specifically, the second loss value can be determined based on the following formula. :

[0085]

[0086] in, For the first The second sub-loss value corresponding to each semantic dimension For the first A set of positive samples with semantic dimensions. For the first The semantic features of the sample image of the nth sample The first sample semantic feature component of each semantic dimension For the first The first semantic dimension of the positive sample set Each positive sample semantic feature component For the first A set of negative samples with semantic dimensions. For the first The th semantic dimension of the negative sample set Each negative sample semantic feature component To compare the interval hyperparameters of the loss, This represents the total number of semantic dimensions.

[0087] In this embodiment, Set to 1. In other implementations, the setting can be adjusted based on specific requirements.

[0088] Step 505: Determine the third loss value based on the sample semantic features and the multiple first sample semantic feature components corresponding to the sample semantic features.

[0089] Specifically, the third loss value can be determined based on the following formula. :

[0090]

[0091] in, For sample images, For diffusion time step, The sample semantic features are obtained after linear transformation of the initial parameter matrix, and include multiple first sample semantic feature components that are aligned one-to-one with multiple semantic dimensions. For hyperparameters, It is standard Gaussian random noise.

[0092] Step 506: Optimize the parameters of the initial encoder based on the first loss value, the second loss value, and the third loss value.

[0093] The parameters of the initial encoder are jointly optimized based on the determined first loss value, second loss value, and third loss value. After reaching the preset number of training rounds or the convergence of each loss function, the training is completed, and the candidate encoder and the candidate parameter matrix of the candidate encoder are obtained.

[0094] In the above scheme, firstly, the initial encoder performs a linear transformation on the sample semantic features based on the initial parameter matrix, generating multiple first sample semantic feature components that correspond one-to-one with multiple semantic dimensions. Then, based on multiple text descriptions corresponding to the semantic dimensions and multiple prompt texts corresponding to the text descriptions, a first mask and a first pseudo-label matching each first sample semantic feature component are generated. A first loss value is calculated based on the first mask, the first pseudo-label, and the first sample semantic feature components. The first loss value constrains the initial encoder and the initial parameter matrix to learn and distinguish semantic information corresponding to different semantic dimensions, ensuring that the output first sample semantic feature components closely match the semantic information of the corresponding semantic dimensions. Then, a second loss value is calculated using the positive sample set, the negative sample set, and the first sample semantic feature components corresponding to each semantic dimension. The second loss value constrains the initial encoder and the initial parameter matrix to increase the distance between positive and negative sample semantic feature components within the same semantic dimension, thereby enhancing the discriminative power of the feature components corresponding to each semantic dimension. Then, a third loss value is calculated by combining the semantic features of the samples with the semantic feature components of the first sample. The third loss value can constrain the semantic feature representation ability of the initial encoder and the initial parameter matrix to adapt to the diffusion process, ensuring that the output sample semantic features can stably participate in diffusion-related operations. Finally, the first, second, and third loss values ​​are combined to optimize all parameters of the initial encoder and adjust the initial parameter matrix synchronously. Under the synchronous effect of multi-dimensional constraints, the candidate encoder and candidate parameter matrix obtained after training can output semantic feature components that are accurately aligned with the semantic dimension and have good discriminative power, as well as output standardized sample semantic features adapted to diffusion calculation.

[0095] An image processing method is also provided in one example of this application, such as Figure 6 As shown, the step of training the initial target dimension predictor and the initial complementary dimension predictor for each semantic dimension based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions includes: Step 601: The candidate encoder is used to determine multiple second sample semantic feature components aligned with multiple semantic dimensions based on the sample semantic features and the candidate parameter matrix.

[0096] The trained candidate encoder performs a linear transformation on the sample semantic features based on the candidate parameter matrix to obtain multiple second sample semantic feature components that are aligned one-to-one with multiple semantic dimensions.

[0097] Step 602: Determine the second mask and second pseudo-label corresponding to each semantic dimension of the sample semantic features based on the multiple second sample semantic feature components corresponding to the sample semantic features, the multiple text descriptions corresponding to the semantic dimensions, and the multiple prompt texts corresponding to the text descriptions.

[0098] Based on the multiple text descriptions corresponding to the semantic dimension and the multiple prompt texts corresponding to each text description, the second mask and the second pseudo-label of the second sample semantic feature component belonging to the semantic dimension can be determined.

[0099] Step 603: Based on the initial target dimension predictor corresponding to each semantic dimension, predict the second sample semantic feature components corresponding to each semantic dimension in the sample semantic features to obtain the first predicted value of each semantic dimension of the second sample semantic features. The input of the initial target dimension predictor is the second sample semantic feature component corresponding to the semantic dimension.

[0100] The second sample semantic feature components corresponding to each semantic dimension are input into the corresponding initial target dimension predictor for prediction, and the first predicted value of the second sample semantic feature corresponding to each semantic dimension is output.

[0101] For example, there are three semantic dimensions: semantic dimension A, semantic dimension B, and semantic dimension C, corresponding to initial target dimension predictors A, B, and C, respectively. The second sample semantic feature component A corresponding to semantic dimension A is input into the initial target dimension predictor A for prediction, resulting in the first predicted value A of the second sample semantic feature component A. The second sample semantic feature component B corresponding to semantic dimension B is input into the initial target dimension predictor B for prediction, resulting in the first predicted value B of the second sample semantic feature component B. The second sample semantic feature component C corresponding to semantic dimension C is input into the initial target dimension predictor C for prediction, resulting in the first predicted value C of the second sample semantic feature component C.

[0102] Step 604: Based on the initial complementary dimension predictor corresponding to each semantic dimension, predict the second sample semantic feature components corresponding to each semantic dimension in the sample semantic features to obtain the second predicted value of each semantic dimension of the second sample semantic features. The input of the initial complementary dimension predictor is the second sample semantic feature components of all semantic dimensions except the corresponding semantic dimension.

[0103] The second sample semantic feature components of all other semantic dimensions, except the second sample semantic feature components of the semantic dimension, are input into the initial target dimension predictor corresponding to that semantic dimension for prediction, and the second predicted value of the second sample semantic feature corresponding to each semantic dimension is output.

[0104] Continuing the example above, semantic dimensions A, B, and C correspond to initial complementary dimension predictors A, B, and C, respectively. The second sample semantic feature components corresponding to all semantic dimensions other than semantic dimension A—namely, second sample semantic feature component B and second sample semantic feature component C—are input into the initial complementary dimension predictor A for prediction, yielding a second predicted value A for second sample semantic feature component A. Similarly, the second sample semantic feature components A and C are input into the initial complementary dimension predictor B for prediction, yielding a second predicted value B for second sample semantic feature component B. Finally, the second sample semantic feature components A and B are input into the initial complementary dimension predictor C for prediction, yielding a second predicted value C for second sample semantic feature component C.

[0105] Step 605: Determine the fourth loss value based on the first predicted value, the second predicted value, the second mask, and the second pseudo-label of each semantic dimension of the sample semantic features.

[0106] Specifically, the fourth loss value can be determined based on the following formula. :

[0107] in, For the first The semantic features of the sample image of the nth sample The second mask corresponding to the second sample semantic feature components of each semantic dimension. For the first The semantic features of the sample image of the nth sample The second pseudo-label corresponds to the second sample semantic feature component of each semantic dimension. For the first The semantic features of the sample image of the nth sample The second sample semantic feature components of each semantic dimension For the first The semantic features of the sample image are related to the first sample image. The complementary semantic dimensions of the first semantic dimension (i.e., excluding the first semantic dimension) The second sample semantic feature component (excluding all other semantic dimensions). For the first An initial target dimension predictor with semantic dimensions, For the first An initial complementary dimension predictor for semantic dimensions, This characterizes the cessation of gradient operations on the semantic features of the samples, meaning that the loss is used only to optimize the parameters of the predictor, freezing the parameters of the encoder.

[0108] Step 606: Optimize the parameters of the initial target dimension predictor and the initial complementary dimension predictor corresponding to each semantic dimension based on the fourth loss value.

[0109] Based on the determined fourth loss value, the parameters of the initial target dimension predictor and the initial complementary dimension predictor corresponding to each semantic dimension are optimized. After reaching the preset number of training rounds or the convergence of each loss function, the training is completed, and the target dimension predictor and the complementary dimension predictor corresponding to each semantic dimension are obtained.

[0110] In the above scheme, firstly, a candidate encoder performs a linear transformation on the sample semantic features based on the candidate parameter matrix, generating multiple second sample semantic feature components that correspond one-to-one with multiple semantic dimensions. Then, based on the multiple text descriptions corresponding to the semantic dimensions and the multiple prompt texts corresponding to each text description, the second mask and second pseudo-label of the second sample semantic feature components corresponding to each semantic dimension are determined. Next, the second sample semantic feature components of the corresponding semantic dimension are input into the initial target dimension predictor to complete the prediction, obtaining the first predicted value for each semantic dimension. Finally, the second sample semantic feature components of the current semantic dimension and the second sample semantic feature components of all other semantic dimensions are input into the initial complementary dimension predictor to complete the prediction, obtaining the second predicted value for each semantic dimension. A fourth loss value is calculated by combining the first predicted value, the second predicted value, the second mask, and the second pseudo-label. This fourth loss value simultaneously constrains the initial target dimension predictor and the initial complementary dimension predictor. The loss term corresponding to the initial target dimension predictor enables it to learn the ability to accurately match the semantic judgment of the corresponding second pseudo-label based on the semantic feature components of the second sample in a single semantic dimension. Similarly, the loss term corresponding to the initial complementary dimension predictor enables it to learn the ability to complete corresponding semantic judgments based on the semantic feature components of the second sample in other complementary semantic dimensions. The parameters of the initial target dimension predictor and the initial complementary dimension predictor for each semantic dimension are optimized using the fourth loss value. Training is completed after reaching a preset number of training epochs or when the loss converges, ultimately yielding the target dimension predictor and the complementary dimension predictor for each semantic dimension.

[0111] An image processing method is also provided in one example of this application, such as Figure 7 As shown, training the candidate encoder based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, multiple prompt texts corresponding to the text descriptions, and the target dimension predictor and complementary dimension predictor corresponding to each semantic dimension includes: Step 701: Based on the target dimension predictor corresponding to each semantic dimension, predict the second sample semantic feature components corresponding to each semantic dimension in the second sample semantic features to obtain the third predicted value of each semantic dimension of the second sample semantic features.

[0112] By using the target dimension predictors corresponding to each semantic dimension that have been trained, the semantic feature components of the second sample corresponding to the same semantic dimension in the semantic features of the sample are predicted, and the third predicted value of each semantic dimension of the semantic features of the second sample is obtained.

[0113] Step 702: Based on the complementary dimension predictor corresponding to each semantic dimension, predict the second sample semantic feature components corresponding to each semantic dimension in the sample semantic features to obtain the fourth predicted value of each semantic dimension of the second sample semantic features.

[0114] By using the complementary dimension predictors corresponding to each semantic dimension that have been trained, the second semantic feature components corresponding to the same semantic dimension in the sample semantic features, as well as the second semantic sample feature components corresponding to all other semantic dimensions except the same semantic dimension, are predicted to obtain the fourth predicted value of each semantic dimension of the second sample semantic feature.

[0115] Step 703: Determine the fifth loss value based on the third predicted value, the fourth predicted value, the second mask, and the second pseudo-label of each semantic dimension of the sample semantic features.

[0116] Specifically, the fifth loss value can be determined based on the following formula. :

[0117] in, For the first The semantic features of the sample image of the nth sample The second mask corresponding to the second sample semantic feature components of each semantic dimension. For the first The semantic features of the sample image of the nth sample The second pseudo-label corresponds to the second sample semantic feature component of each semantic dimension. For the first The semantic features of the sample image of the nth sample The second sample semantic feature components of each semantic dimension For the first A target dimension predictor with one semantic dimension. For the first A complementary dimension predictor for semantic dimensions. For the first Sample semantic features of a sample image.

[0118] Step 704: Determine the first loss value again based on the multiple second sample semantic feature components of the sample semantic features and the second mask and second pseudo-label corresponding to each semantic dimension.

[0119] Specifically, the first loss value can be re-determined based on the following formula. :

[0120] in, For the first The semantic features of the sample image of the nth sample The second mask corresponding to the second sample semantic feature components of each semantic dimension. For the first The semantic features of the sample image of the nth sample The second pseudo-label corresponds to the second sample semantic feature component of each semantic dimension. For the first The semantic features of the sample image of the nth sample The second sample semantic feature component of each semantic dimension.

[0121] Step 705: Redetermine the second loss value based on the multiple second sample semantic feature components, the positive sample set, and the negative sample set of the sample semantic features.

[0122] Specifically, the second loss value can be re-determined based on the following formula. :

[0123]

[0124] in, For the first The second sub-loss value corresponding to each semantic dimension For the first A set of positive samples with semantic dimensions. For the first The semantic features of the sample image of the nth sample The second sample semantic feature components of each semantic dimension For the first The first semantic dimension of the positive sample set Each positive sample semantic feature component For the first A set of negative samples with semantic dimensions. For the first The th semantic dimension of the negative sample set Each negative sample semantic feature component To compare the interval hyperparameters of the loss, This represents the total number of semantic dimensions.

[0125] In this embodiment, Set to 1. In other implementations, the setting can be adjusted based on specific requirements.

[0126] Step 706: Redetermine the third loss value based on the semantic features of the sample.

[0127] Specifically, the third loss value can be re-determined based on the following formula. :

[0128]

[0129] in, For sample images, For diffusion time step, The sample semantic features are obtained after linear transformation of the candidate parameter matrix, which includes multiple second sample semantic feature components aligned one-to-one with multiple semantic dimensions. For hyperparameters, It is standard Gaussian random noise.

[0130] Step 707: Optimize the parameters of the candidate encoder based on the first loss value, the second loss value, the third loss value, and the fifth loss value.

[0131] Based on the determined fifth loss value, the first, second, and third loss values ​​are redefined to jointly optimize the parameters of the candidate encoder. After reaching the preset number of training rounds or when each loss function converges, the training is completed, and the encoder and its parameter matrix are obtained.

[0132] In the above scheme, the second sample semantic feature components of the corresponding semantic dimension are first input into the trained target dimension predictor for prediction, obtaining the first predicted value for each semantic dimension. Then, the second sample semantic feature components of the current semantic dimension, as well as the second sample semantic feature components of all other semantic dimensions, are input into the trained complementary dimension predictor for prediction, obtaining the second predicted value for each semantic dimension. The fifth loss value is calculated by combining the third and fourth predicted values, the second mask, and the second pseudo-label. This fifth loss value constrains the candidate encoder, enabling the second sample semantic feature components output by the candidate encoder to learn the semantic judgment effect of the target dimension predictor and weaken the semantic interference brought by the complementary dimension predictor, thereby strengthening the specific expressive ability of each semantic dimension's own features. The first, second, and third loss values ​​are recalculated based on the semantic feature components of the second sample. The fifth loss value, the recalculated first, second, and third loss values ​​are then used to jointly optimize the parameters of the candidate encoder. Training is completed when the preset number of training rounds is reached or the loss converges. Finally, the encoder and the parameter matrix inside the encoder are obtained. The semantic feature components of the second sample output by the encoder can satisfy the semantic judgment requirements of the target dimension predictor and the complementary dimension predictor, and can also take into account the semantic discrimination effect, thereby significantly improving the fidelity of image reconstruction through the encoder.

[0133] An image processing method is also provided in one example of this application, such as Figure 8 As shown, determining the mask and pseudo-label corresponding to each semantic dimension of the sample semantic features includes: Step 801: Determine the feature semantic features of each feature category of the semantic dimension based on the multiple text descriptions corresponding to the semantic dimension and the multiple prompt texts corresponding to the text descriptions. The feature categories include mutually exclusive first feature categories and second feature categories, as well as a third feature category that does not belong to either the first feature category or the second feature category.

[0134] In this embodiment, the feature categories include three categories: A (first feature category), B (second feature category), and O (other categories). Among them, A and B are mutually exclusive feature categories, while O is a feature category that belongs to neither A nor B.

[0135] For example, the semantic dimension is "hat," which includes three text descriptions: "wearing a hat," "not wearing a hat," and "baseball cap." If "wearing a hat" belongs to A, then "not wearing a hat" belongs to B. "Baseball cap" belongs to neither A nor B, therefore, "baseball cap" belongs to O.

[0136] Step 802: Based on the similarity between each sample semantic feature component of the sample semantic feature and the category semantic features of each feature category of the corresponding semantic dimension of the sample semantic feature component.

[0137] Specifically, the number can be determined based on the following formula. Category semantic features of each feature category in each semantic dimension :

[0138] in, For the first Each semantic dimension belongs to the feature category. The total number of all prompt texts corresponding to all text descriptions. For the first Each semantic dimension belongs to the feature category. The text vector of the prompt text corresponding to all text descriptions.

[0139] The text descriptions under the semantic dimension are divided into three sets according to feature categories: the set belonging to A, the set belonging to B, and the set belonging to C. The text vectors of all prompt texts corresponding to all text descriptions in each of these three sets are averaged and normalized to obtain the category semantic features of the corresponding three categories under this semantic dimension.

[0140] Furthermore, it is possible to calculate the cosine similarity between each sample semantic feature component of the sample semantic features and the category semantic features of the three categories in the corresponding semantic dimension.

[0141] For example, there are three semantic dimensions: semantic dimension A, semantic dimension B, and semantic dimension C. Semantic dimension A has a category semantic feature A corresponding to feature category A, a category semantic feature B corresponding to feature category B, and a category semantic feature C corresponding to feature category C. Semantic dimension B has a category semantic feature D corresponding to feature category A, a category semantic feature E corresponding to feature category B, and a category semantic feature F corresponding to feature category C. Semantic dimension C has a category semantic feature G corresponding to feature category A, a category semantic feature H corresponding to feature category B, and a category semantic feature I corresponding to feature category C. The sample semantic feature component A corresponding to semantic dimension A is calculated with category semantic feature A to obtain similarity A; the cosine similarity is calculated with category semantic feature B to obtain similarity B; and the cosine similarity is calculated with category semantic feature C to obtain similarity C. For each sample semantic feature, calculate the cosine similarity between the sample semantic feature component B corresponding to semantic dimension B and the category semantic feature D to obtain similarity D; calculate the cosine similarity between the sample semantic feature component B and the category semantic feature E to obtain similarity E; and calculate the cosine similarity between the sample semantic feature component C and the category semantic feature G to obtain similarity G; calculate the cosine similarity between the sample semantic feature component C and the category semantic feature H to obtain similarity H; and calculate the cosine similarity between the sample semantic feature component C and the category semantic feature I to obtain similarity I.

[0142] Step 803: Determine the feature category with the highest similarity as the feature category to which the semantic feature component of the sample belongs.

[0143] Continuing with the previous example, among the similarities corresponding to sample semantic feature component A, similarity B is the highest; therefore, the feature category to which sample semantic feature component A belongs is determined to be feature category B. Among the similarities corresponding to sample semantic feature component B, similarity D is the highest; therefore, the feature category to which sample semantic feature component B belongs is determined to be feature category A. Among the similarities corresponding to sample semantic feature component C, similarity I is the highest; therefore, the feature category to which sample semantic feature component C belongs is determined to be feature category C.

[0144] Step 804: Determine the mask and pseudo-label of the semantic dimension corresponding to the semantic feature of the sample based on the feature category to which the sample semantic feature component belongs.

[0145] Specifically, the number can be determined based on the following formula. The semantic features corresponding to the first sample A mask with semantic dimensions With pseudo-tags :

[0146]

[0147] in, For the first The semantic features corresponding to the first sample The feature category to which the semantic feature components of each semantic dimension of the sample belong. For feature category A, The feature category is B.

[0148] Continuing with the previous example, sample semantic feature component A belongs to feature category B. Therefore, the mask for its semantic feature in semantic dimension A is 1, and the pseudo-label is 0. Sample semantic feature component B belongs to feature category A. Therefore, the mask for its semantic feature in semantic dimension B is 1, and the pseudo-label is 1. Sample semantic feature component C belongs to feature category C. Therefore, the mask for its semantic feature in semantic dimension C is 0, and the pseudo-label is 0.

[0149] In the above scheme, firstly, based on the multiple text descriptions corresponding to the semantic dimension and the multiple prompt texts corresponding to each text description, the category semantic features of each feature category under the semantic dimension are determined. Then, the similarity between each sample semantic feature component and the category semantic features of each feature category in the corresponding semantic dimension is calculated. The feature category with the highest similarity is determined as the feature category to which the sample semantic feature component belongs. Finally, based on the feature category to which the sample semantic feature component belongs, the mask and pseudo-label of the semantic dimension corresponding to the sample semantic feature are determined. The distinction between masks and pseudo-labels based on feature categories stems from the fact that the first and second feature categories have clear semantic boundaries and are mutually exclusive, possessing explicit semantic judgment value. Setting the corresponding sample semantic feature components of these two categories as valid masks allows the corresponding samples to participate in subsequent training loss calculations, providing stable and clear semantic supervision signals for encoder training. The semantic state corresponding to the third feature category is ambiguous and lacks explicit classification supervision value. Setting it as an invalid mask avoids introducing noise interference into the training process due to ambiguous semantics, improving training stability and the accuracy of semantic judgment. At the same time, determining the pseudo-label value based on whether the sample semantic feature component belongs to the first feature category can correspond to the two mutually exclusive semantic states, providing corresponding label data for classification loss calculation.

[0150] An image processing method is also provided in one example of this application, such as Figure 9 As shown, the method further includes: Step 901: Determine the variance of the semantic dimension based on the semantic feature components of the samples in the training sample set whose feature categories are the first feature category and the second feature category.

[0151] For each semantic dimension, the variance of that semantic dimension is determined based on the semantic feature components of the samples in the training sample set whose corresponding feature categories are the first and second feature categories.

[0152] Step 902: If the variance is less than a preset variance, then determine the frequency of occurrence of the semantic dimension in all positive sample sets in the training sample set.

[0153] In this embodiment, the preset variance is set to a value in the range of 0.1-0.2. In other embodiments, it can be set according to specific needs.

[0154] If the variance of a certain semantic dimension is less than a preset variance, then the probability of that semantic dimension appearing in all positive samples in the training sample set is calculated. That is, the total number of semantic feature components corresponding to that semantic dimension in all positive samples is divided by the total number of semantic feature components of all samples in all positive samples.

[0155] Step 903: Determine the correction weight of the semantic dimension based on the occurrence frequency. The correction weight is used to correct the sample semantic feature components belonging to the semantic dimension when determining the first loss value and the second loss value.

[0156] Specifically, the number can be determined based on the following formula. Correction weights for each semantic dimension :

[0157] in, For the first The probability of occurrence of each semantic dimension It should be a very small positive number to prevent the denominator from being 0.

[0158] In the above scheme, the variance of each semantic dimension is calculated based on the semantic feature components of samples belonging to the first and second feature categories in the training sample set. Then, for semantic dimensions with variances less than a preset variance, their frequency of occurrence in all positive samples in the training sample set is statistically analyzed. Finally, the correction weight of the corresponding semantic dimension is determined based on the frequency of occurrence. The correction weight is used to correct the semantic feature components of the samples belonging to the corresponding semantic dimension when determining the first and second loss values. A smaller variance indicates that the semantic dimension has a lower or higher proportion in the training sample set. By statistically analyzing its frequency of occurrence in the positive sample set and determining the corresponding correction weight, the influence of semantic dimensions with concentrated feature distributions and unbalanced sample proportions in loss calculation can be specifically adjusted. This balances the contribution of different semantic dimensions to the training process, avoids some dimensions interfering with the stability of loss calculation due to excessive differences in sample distribution, and makes the calculation results of the first and second loss values ​​more balanced and reasonable, thereby improving the overall effect of encoder training.

[0159] To implement the above image processing methods, such as Figure 10 As shown, an example of this application provides an image processing apparatus, including: The processing module 1001 is used to extract features from the original image input by the user to obtain semantic features; The calculation module 1002 is used to determine multiple semantic feature components and multiple residual feature components aligned with multiple semantic dimensions using the encoder based on the semantic features and the parameter matrix. The parameter matrix is ​​obtained by training the encoder using a training sample set. In the parameter matrix, all elements on the diagonal are constrained to be positive and all elements above the diagonal are constrained to be zero. The multiple semantic dimensions are extracted from the training sample set. The semantic feature components include the semantic information of the original image in the corresponding semantic dimension, and the residual feature components include the semantic information of the original image outside the multiple semantic dimensions. The generation module 1003 is used to determine the target image based on the semantic features, the plurality of semantic feature components, and the plurality of residual feature components.

[0160] The generation module 1003 is further configured to perform forward diffusion on the semantic features to obtain noisy semantic features; The generation module 1003 is further configured to perform reverse diffusion on the noisy semantic features based on the plurality of semantic feature components and the plurality of residual feature components to obtain the target image.

[0161] The processing module 1001 is further configured to use a language model to extract semantics from the training sample set to obtain multiple initial semantic dimensions and multiple text descriptions corresponding to the initial semantic dimensions. The processing module 1001 is further configured to use a language model to generate multiple semantically identical prompt texts corresponding to each text description of the initial semantic dimension; The calculation module 1002 is further configured to determine the total coverage of the initial semantic dimension in the training sample set based on the multiple text descriptions corresponding to the initial semantic dimension; The calculation module 1002 is also used to determine the semantic similarity between each text description corresponding to the initial semantic dimension and each verification sample image in the verification sample set. The calculation module 1002 is further configured to determine the mean semantic similarity difference of the initial semantic dimension based on the highest and second-highest semantic similarity between each verification sample image and each text description; The calculation module 1002 is further configured to determine multiple initial semantic dimensions that meet the target conditions as the multiple semantic dimensions, wherein the target conditions are that the total coverage of the initial semantic dimensions belongs to a preset coverage range, the initial semantic dimensions have at least two text descriptions with coverage greater than the preset coverage, and the average semantic similarity difference of the initial semantic dimensions is greater than a preset difference.

[0162] The processing module 1001 is further configured to extract features from each sample image in the training sample set to obtain corresponding sample semantic features. The calculation module 1002 is further configured to train the initial encoder based on the sample semantic features, the initial parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions, so as to obtain the candidate encoder and the candidate parameter matrix. The calculation module 1002 is further configured to train the initial target dimension predictor and the initial complementary dimension predictor for each semantic dimension based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension and multiple prompt texts corresponding to the text descriptions, so as to obtain the target dimension predictor and the complementary dimension predictor for each semantic dimension. The calculation module 1002 is further configured to train the candidate encoder based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, multiple prompt texts corresponding to the text descriptions, and the target dimension predictor and complementary dimension predictor corresponding to each semantic dimension, so as to obtain the encoder and the parameter matrix.

[0163] The calculation module 1002 is further configured to use an initial encoder to determine multiple first sample semantic feature components aligned with multiple semantic dimensions based on the sample semantic features and the initial parameter matrix, wherein the sample semantic feature components include the semantic information of the sample image in the corresponding semantic dimension. The calculation module 1002 is further configured to determine the first mask and the first pseudo-label corresponding to each semantic dimension of the sample semantic features based on the multiple first sample semantic feature components of the sample semantic features, the multiple text descriptions corresponding to the semantic dimensions, and the multiple prompt texts corresponding to the text descriptions. The calculation module 1002 is further configured to determine a first loss value based on multiple first sample semantic feature components of the sample semantic features and the first mask and first pseudo label corresponding to each semantic dimension. The calculation module 1002 is further configured to determine a second loss value based on multiple first sample semantic feature components of the sample semantic features, and positive sample sets and negative sample sets corresponding to each semantic dimension. The positive sample set includes multiple positive sample semantic feature components corresponding to the semantic dimension, and the negative sample set includes multiple negative sample semantic feature components corresponding to the semantic dimension. The calculation module 1002 is further configured to determine a third loss value based on the sample semantic features and multiple first sample semantic feature components corresponding to the sample semantic features; The processing module 1001 is further configured to optimize the parameters of the initial encoder based on the first loss value, the second loss value, and the third loss value.

[0164] The calculation module 1002 is further configured to use the candidate encoder to determine multiple second sample semantic feature components aligned with multiple semantic dimensions based on the sample semantic features and the candidate parameter matrix. The calculation module 1002 is further configured to determine the second mask and the second pseudo-label corresponding to each semantic dimension of the sample semantic features based on the multiple second sample semantic feature components corresponding to the sample semantic features, the multiple text descriptions corresponding to the semantic dimensions, and the multiple prompt texts corresponding to the text descriptions. The calculation module 1002 is further configured to predict the second sample semantic feature components corresponding to each semantic dimension in the sample semantic features according to the initial target dimension predictor corresponding to each semantic dimension, so as to obtain the first predicted value of each semantic dimension of the second sample semantic features. The input of the initial target dimension predictor is the second sample semantic feature component corresponding to the semantic dimension. The calculation module 1002 is further configured to predict the second sample semantic feature components corresponding to each semantic dimension in the sample semantic features according to the initial complementary dimension predictor corresponding to each semantic dimension, so as to obtain the second predicted value of each semantic dimension of the second sample semantic features. The input of the initial complementary dimension predictor is the second sample semantic feature components of all semantic dimensions except the corresponding semantic dimension. The calculation module 1002 is further configured to determine a fourth loss value based on the first predicted value, the second predicted value, the second mask, and the second pseudo-label of each semantic dimension of the sample semantic features. The processing module 1001 is further configured to optimize the parameters of the initial target dimension predictor and the initial complementary dimension predictor corresponding to each semantic dimension based on the fourth loss value.

[0165] The calculation module 1002 is further configured to predict the second sample semantic feature components corresponding to each semantic dimension in the second sample semantic features according to the target dimension predictor corresponding to each semantic dimension, so as to obtain the third predicted value of each semantic dimension of the second sample semantic features. The calculation module 1002 is further configured to predict the second sample semantic feature components corresponding to each semantic dimension in the sample semantic features according to the complementary dimension predictor corresponding to each semantic dimension, so as to obtain the fourth predicted value of each semantic dimension of the second sample semantic features. The calculation module 1002 is also used to determine a fifth loss value based on the third predicted value, the fourth predicted value, the second mask, and the second pseudo-label of each semantic dimension of the sample semantic features. The calculation module 1002 is further configured to redetermine the first loss value based on the multiple second sample semantic feature components of the sample semantic features and the second mask and second pseudo-label corresponding to each semantic dimension. The calculation module 1002 is further configured to redetermine the second loss value based on the multiple second sample semantic feature components, the positive sample set, and the negative sample set of the sample semantic features; The calculation module 1002 is also used to redetermine the third loss value based on the semantic features of the sample; The processing module 1001 is further configured to optimize the parameters of the candidate encoder based on the first loss value, the second loss value, the third loss value, and the fifth loss value.

[0166] The calculation module 1002 is further configured to determine the feature semantic features of each feature category of the semantic dimension based on the multiple text descriptions corresponding to the semantic dimension and the multiple prompt texts corresponding to the text descriptions. The feature categories include mutually exclusive first feature categories and second feature categories, as well as a third feature category that does not belong to either the first feature category or the second feature category. The calculation module 1002 is further configured to calculate the similarity between the sample semantic feature components and the category semantic features of each feature category of the semantic dimension corresponding to the sample semantic feature components. The calculation module 1002 is also used to determine the feature category with the highest similarity as the feature category to which the semantic feature component of the sample belongs; The calculation module 1002 is further configured to determine the mask and pseudo-label of the semantic dimension corresponding to the semantic feature of the sample based on the feature category to which the sample semantic feature component belongs.

[0167] The calculation module 1002 is further configured to determine the variance of the semantic dimension based on the semantic feature components of the samples in the training sample set whose feature categories are the first feature category and the second feature category. The calculation module 1002 is further configured to determine the frequency of occurrence of the semantic dimension in all positive sample sets in the training sample set if the variance is less than a preset variance. The calculation module 1002 is further configured to determine the correction weight of the semantic dimension based on the occurrence frequency, and the correction weight is used to correct the sample semantic feature components belonging to the semantic dimension when determining the first loss value and the second loss value.

[0168] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0169] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0170] like Figure 11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded into a random access memory (RAM) 1103 from a storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of the electronic device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0171] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of displays, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0172] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the image processing method by any other suitable means (e.g., by means of firmware).

[0173] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0175] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0177] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0178] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0179] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0180] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0181] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. An image processing method, the method comprising: Semantic features are obtained by extracting features from the original image input by the user. An encoder is used to determine multiple semantic feature components and multiple residual feature components aligned with multiple semantic dimensions based on the semantic features and parameter matrix. The parameter matrix is ​​obtained by training the encoder using a training sample set. In the parameter matrix, all elements on the diagonal are constrained to be positive and all elements above the diagonal are constrained to be zero. The multiple semantic dimensions are extracted from the training sample set. The semantic feature components include the semantic information of the original image in the corresponding semantic dimension, and the residual feature components include the semantic information of the original image outside the multiple semantic dimensions. The target image is determined based on the semantic features, the plurality of semantic feature components, and the plurality of residual feature components.

2. The method according to claim 1, wherein determining the target image based on the semantic features, the plurality of semantic feature components, and the plurality of residual feature components comprises: The semantic features are forward diffused to obtain noisy semantic features; The noisy semantic features are back-diffused based on the multiple semantic feature components and the multiple residual feature components to obtain the target image.

3. The method according to claim 1, further comprising: The training sample set is semantically extracted using a language model to obtain multiple initial semantic dimensions and multiple text descriptions corresponding to the initial semantic dimensions. The language model is used to generate multiple semantically identical prompt texts corresponding to each text description in the initial semantic dimension; The total coverage of the initial semantic dimension in the training sample set is determined based on multiple text descriptions corresponding to the initial semantic dimension. Determine the semantic similarity between each text description corresponding to the initial semantic dimension and each verification sample image in the verification sample set; The mean semantic similarity difference of the initial semantic dimension is determined based on the highest and second-highest semantic similarity between each verification sample image and each text description; Multiple initial semantic dimensions that meet the target conditions are determined as the multiple semantic dimensions. The target conditions are that the total coverage of the initial semantic dimensions is within a preset coverage range, the initial semantic dimensions have at least two text descriptions with coverage greater than the preset coverage, and the average semantic similarity difference of the initial semantic dimensions is greater than a preset difference.

4. The method according to claim 3, further comprising: Feature extraction is performed on each sample image in the training sample set to obtain the corresponding sample semantic features; The initial encoder is trained based on the sample semantic features, the initial parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions to obtain the candidate encoder and the candidate parameter matrix. Based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions, the initial target dimension predictor and the initial complementary dimension predictor corresponding to each semantic dimension are trained to obtain the target dimension predictor and the complementary dimension predictor corresponding to each semantic dimension. The candidate encoder is trained based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, multiple prompt texts corresponding to the text descriptions, and the target dimension predictor and complementary dimension predictor corresponding to each semantic dimension, to obtain the encoder and the parameter matrix.

5. The method according to claim 4, wherein training the initial encoder based on the sample semantic features, the initial parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions includes: The initial encoder determines multiple first sample semantic feature components aligned with multiple semantic dimensions based on the sample semantic features and the initial parameter matrix. The sample semantic feature components include the semantic information of the sample image in the corresponding semantic dimension. The first mask and the first pseudo-label corresponding to each semantic dimension of the sample semantic features are determined based on multiple first sample semantic feature components of the sample semantic features, multiple text descriptions corresponding to the semantic dimensions, and multiple prompt texts corresponding to the text descriptions. The first loss value is determined based on the multiple first sample semantic feature components of the sample semantic features and the first mask and first pseudo label corresponding to each semantic dimension; The second loss value is determined based on multiple first sample semantic feature components of the sample semantic features, the positive sample set and the negative sample set corresponding to each semantic dimension. The positive sample set includes multiple positive sample semantic feature components corresponding to the semantic dimension, and the negative sample set includes multiple negative sample semantic feature components corresponding to the semantic dimension. A third loss value is determined based on the sample semantic features and multiple first sample semantic feature components corresponding to the sample semantic features; The parameters of the initial encoder are optimized based on the first loss value, the second loss value, and the third loss value.

6. The method according to claim 5, wherein training the initial target dimension predictor and the initial complementary dimension predictor for each semantic dimension based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, and multiple prompt texts corresponding to the text descriptions, comprises: The candidate encoder uses the sample semantic features and the candidate parameter matrix to determine multiple second sample semantic feature components aligned with multiple semantic dimensions; The second mask and the second pseudo-label corresponding to each semantic dimension of the sample semantic feature are determined based on the multiple second sample semantic feature components corresponding to the sample semantic feature, the multiple text descriptions corresponding to the semantic dimension, and the multiple prompt texts corresponding to the text descriptions. The initial target dimension predictor predicts the second sample semantic feature components corresponding to each semantic dimension in the sample semantic features according to the initial target dimension predictor, and obtains the first predicted value of each semantic dimension of the second sample semantic features. The input of the initial target dimension predictor is the second sample semantic feature component corresponding to the semantic dimension. The second sample semantic feature components corresponding to each semantic dimension in the sample semantic features are predicted according to the initial complementary dimension predictor corresponding to each semantic dimension, so as to obtain the second predicted value of each semantic dimension of the second sample semantic features. The input of the initial complementary dimension predictor is the second sample semantic feature components of all semantic dimensions except the corresponding semantic dimension. The fourth loss value is determined based on the first predicted value, the second predicted value, the second mask, and the second pseudo-label of each semantic dimension of the sample semantic features; The parameters of the initial target dimension predictor and the initial complementary dimension predictor corresponding to each semantic dimension are optimized based on the fourth loss value.

7. The method according to claim 6, wherein training the candidate encoder based on the sample semantic features, the candidate parameter matrix, multiple text descriptions corresponding to each semantic dimension, multiple prompt texts corresponding to the text descriptions, and the target dimension predictor and complementary dimension predictor corresponding to each semantic dimension includes: Based on the target dimension predictor corresponding to each semantic dimension, the second sample semantic feature components corresponding to each semantic dimension in the second sample semantic feature are predicted to obtain the third predicted value of each semantic dimension of the second sample semantic feature. Based on the complementary dimension predictor corresponding to each semantic dimension, the second sample semantic feature components corresponding to each semantic dimension in the sample semantic features are predicted to obtain the fourth predicted value of each semantic dimension of the second sample semantic features. The fifth loss value is determined based on the third predicted value, the fourth predicted value, the second mask, and the second pseudo-label of each semantic dimension of the sample semantic features; The first loss value is re-determined based on the multiple second sample semantic feature components of the sample semantic features and the second mask and second pseudo-label corresponding to each semantic dimension; The second loss value is re-determined based on multiple second sample semantic feature components, the positive sample set, and the negative sample set of the sample semantic features; The third loss value is re-determined based on the semantic features of the sample; The parameters of the candidate encoder are optimized based on the first loss value, the second loss value, the third loss value, and the fifth loss value.

8. The method according to claim 5, 6, or 7, determining the mask and pseudo-label corresponding to each semantic dimension of the sample semantic features, comprising: Based on multiple text descriptions corresponding to the semantic dimension and multiple prompt texts corresponding to the text descriptions, the semantic features of each feature category of the semantic dimension are determined. The feature categories include a first feature category and a second feature category that are mutually exclusive, and a third feature category that does not belong to either the first feature category or the second feature category. Based on the similarity between each sample semantic feature component of the sample semantic feature and the category semantic features of each feature category corresponding to the semantic dimension of the sample semantic feature component; The feature category with the highest similarity is determined as the feature category to which the semantic feature component of the sample belongs; The mask and pseudo-label of the semantic dimension corresponding to the semantic feature of the sample are determined based on the feature category to which the sample semantic feature component belongs.

9. The method according to claim 8, further comprising: The variance of the semantic dimension is determined based on the semantic feature components of the samples in the training sample set whose feature categories are the first feature category and the second feature category. If the variance is less than a preset variance, then the frequency of occurrence of the semantic dimension in all positive sample sets in the training sample set is determined; The correction weight of the semantic dimension is determined based on the occurrence frequency. The correction weight is used to correct the sample semantic feature components belonging to the semantic dimension when determining the first loss value and the second loss value.

10. An image processing apparatus, the apparatus comprising: The processing module is used to extract features from the original image input by the user to obtain semantic features; The calculation module is used to determine multiple semantic feature components and multiple residual feature components aligned with multiple semantic dimensions using the encoder based on the semantic features and parameter matrix. The parameter matrix is ​​obtained by training the encoder using a training sample set. In the parameter matrix, all elements on the diagonal are constrained to be positive and all elements above the diagonal are constrained to be zero. The multiple semantic dimensions are extracted from the training sample set. The semantic feature components include the semantic information of the original image in the corresponding semantic dimension, and the residual feature components include the semantic information of the original image outside the multiple semantic dimensions. The generation module is used to determine the target image based on the semantic features, the plurality of semantic feature components, and the plurality of residual feature components.

Images