Image processing method and apparatus

By updating the image processing model in the multimodal processing model, using the text label matching results of the target task and image samples, the problem of cross-modal matching of the multimodal processing model in downstream tasks is solved, and the processing performance is improved.

WO2024232835A1PCT designated stage expired Publication Date: 2024-11-14ALIBABA INNOVATION PRIVATE LIMITED

Patent Information

Application Number
PCT/SG2024/050310
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-05-10
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Multimodal processing models are difficult to achieve cross-modal matching between images and text in downstream tasks, resulting in poor performance, mainly due to the differences in training data distribution in pre-training and downstream tasks.

Method used

By determining the image to be processed and inputting it into the target image processing model, the initial image processing model is updated according to the matching results of at least two text labels corresponding to the target task and image samples, and cross-modal matching between the image and text is achieved, and multiple improvements are improved. Application performance of modal processing models.

Benefits of technology

This method enables the initial image processing model to learn training data information in the target task, improves the performance of the target image processing model, realizes cross-modal matching between images and text, and improves the multimodal processing model to be applied to the target Processing performance during tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SG2024050310_14112024_PF_FP_ABST
    Figure SG2024050310_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure provide an image processing method and apparatus. The image processing method comprises: determining an image to be processed; and inputting said image into a target image processing model, and obtaining target text information of said image, wherein the target image processing model is obtained by updating an initial image processing model according to an image sample determined by a target task and a matching result of at least two text tags corresponding to the image sample.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-Reference to Image Processing Method and Apparatus This disclosure claims priority to Chinese patent application No. 202310536807.6, filed with the China Patent Office on May 10, 2023, entitled "Image Processing Method and Apparatus," the entire contents of which are incorporated herein by reference. Technical Field: Embodiments of the present disclosure relate to the field of computer technology, and more particularly, to image processing methods. Background: Multimodal processing models can be configured to process various types of data, such as text and images. Modalities can be understood as different ways of representing data, such as text and images being different modalities. Multimodal processing models can convert input from one modality into output from another modality. Typically, multimodal processing models can be pre-trained using large-scale datasets to impart general capabilities for image and text data processing. However, due to distribution differences between the training data used in pre-training and the training data used in downstream tasks, cross-modal matching between images and text is difficult to achieve when the multimodal processing model is used for downstream tasks, resulting in poor performance of the multimodal processing model when applied to downstream tasks. Therefore, an effective technical solution is urgently needed to address the above-mentioned issues. In view of this, embodiments of the present disclosure provide an image processing method. One or more embodiments of the present disclosure also relate to an image processing device, another image processing method, another image processing device, an image processing model training method, an image processing model training device, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art. According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, comprising: determining an image to be processed; inputting the image to be processed into a target image processing model to obtain target text information for the image to be processed, wherein the target image processing model is obtained by updating an initial image processing model based on an image sample determined according to a target task and a matching result between at least two text labels corresponding to the image sample. According to a second aspect of an embodiment of the present disclosure, an image processing apparatus is provided, comprising: a determination module configured to determine an image to be processed; and an input module configured to input the image to be processed into a target image processing model to obtain target text information of the image to be processed, wherein the target image processing model is obtained by updating an initial image processing model based on an image sample determined according to a target task and a matching result of at least two text labels corresponding to the image sample.According to a third aspect of an embodiment of the present disclosure, another image processing method is provided, which is applied to a cloud-side device and includes: receiving an image processing request sent by an end-side device, wherein the image processing request carries an image to be processed; inputting the image to be processed into a target image processing model to obtain target text information of the image to be processed, wherein the target image processing model is obtained by updating an initial image processing model based on an image sample determined according to a target task and a matching result of at least two text labels corresponding to the image sample; and sending the target text information to the end-side device. According to a fourth aspect of an embodiment of the present disclosure, another image processing apparatus is provided, which is applied to a cloud-side device and includes: a receiving module, configured to receive an image processing request sent by an end-side device, wherein the image processing request carries an image to be processed; an input module, configured to input the image to be processed into a target image processing model to obtain target text information of the image to be processed, wherein the target image processing model is obtained by updating an initial image processing model based on an image sample determined according to a target task and a matching result of at least two text labels corresponding to the image sample; and a sending module, configured to send the target text information to the end-side device. According to a fifth aspect of an embodiment of the present disclosure, a method for training an image processing model is provided, comprising: determining an image sample and at least two text labels corresponding to the image sample according to a target task; inputting the image sample and the at least two text labels into an initial image processing model; determining, in the initial image processing model, initial image features corresponding to the image sample and target text features corresponding to the text labels; processing the initial image features according to the target text features corresponding to the text labels to obtain target image features corresponding to the image sample; and updating the initial image processing model according to a matching result between the target image features and the target text features corresponding to the text labels to obtain a target image processing model.According to a sixth aspect of an embodiment of the present disclosure, an image processing model training apparatus is provided, comprising: a first determination module configured to determine an image sample and at least two text labels corresponding to the image sample based on a target task; an input module configured to input the image sample and the at least two text labels into an initial image processing model; a second determination module configured to determine, in the initial image processing model, initial image features corresponding to the image sample and target text features corresponding to the text labels; a processing module configured to process the initial image features based on the target text features corresponding to the text labels to obtain target image features corresponding to the image sample; and an update module configured to update the initial image processing model based on matching results between the target image features and the target text features corresponding to the text labels to obtain a target image processing model. According to a seventh aspect of an embodiment of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the steps of the above-described method. According to an eighth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, storing computer-executable instructions. When executed by a processor, the instructions implement the steps of the above-described method. According to a ninth aspect of an embodiment of the present disclosure, a computer program is provided. When executed on a computer, the computer is caused to execute the steps of the above-described method. One embodiment of the present disclosure provides an image processing method, comprising: determining an image to be processed; inputting the image to be processed into a target image processing model; and obtaining target text information for the image to be processed. The target image processing model is obtained by updating an initial image processing model based on image samples determined for a target task and matching results of at least two text labels corresponding to the image samples. In the above method, the target image processing model configured to process the image is obtained by updating the initial image processing model based on the image samples and text labels determined for the target task. This enables the initial image processing model to learn information from the training data for the target task, thereby ensuring the performance of the updated target image processing model in processing the target task.Moreover, the updating of the initial image processing model is achieved based on the matching results between the image sample and at least two text labels corresponding to the image sample, which enables the initial image processing model to learn information about multiple discriminative text labels and realize cross-modal matching between image and text based on the matching between the image sample and at least two text labels, thereby improving the processing performance of the multimodal processing model when applied to the target task. BRIEF DESCRIPTION OF THE DRAWINGS FIG1 is a schematic diagram of an application scenario of an image processing method provided by one embodiment of the present disclosure; FIG2 is a flowchart of an image processing method provided by one embodiment of the present disclosure; FIG3 is a flowchart of updating a target image processing model in an image processing method provided by one embodiment of the present disclosure; FIG4 is a flowchart of the training process of an image processing method provided by one embodiment of the present disclosure; FIG5 is a flowchart of the processing process of an image processing method provided by one embodiment of the present disclosure; FIG6 is a schematic diagram of the structure of an image processing device provided by one embodiment of the present disclosure; FIG7 is a flowchart of another image processing method provided by one embodiment of the present disclosure; FIG8 is a schematic diagram of the structure of another image processing device provided by one embodiment of the present disclosure; FIG9 is a schematic diagram of the structure of an image processing model training device provided by one embodiment of the present disclosure; FIG10 is a block diagram of the structure of a computing device provided by one embodiment of the present disclosure. DETAILED DESCRIPTION The following description sets forth numerous specific details to facilitate a thorough understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art may make similar generalizations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific implementations disclosed below. The terminology used in one or more embodiments of the present disclosure is for the purpose of describing specific embodiments and is not intended to limit the present disclosure. As used in one or more embodiments of the present disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should be understood that while the terms first, second, etc. may be employed in one or more embodiments of the present disclosure to describe various information, such information should not be limited to these terms. These terms are used to distinguish information of the same type from one another. For example, first could be referred to as second, and similarly, second could be referred to as first, without departing from the scope of one or more embodiments of the present disclosure. Depending on the context, the term "if," as used herein, could be interpreted as meaning "when," "when," or "in response to determining."Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data set for analysis, stored data, and displayed data, etc.) involved in one or more embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for the user to choose to authorize or refuse. First, the terms involved in one or more embodiments of the present disclosure are explained. Zero-shot recognition: Zero-shot recognition refers to the transfer of cross-modal knowledge through the use of semantic information, thereby learning to recognize unseen image categories using visible class information. Combinatorial zero-shot recognition: Combinatorial zero-shot recognition is a special case of the zero-shot recognition task. It refers to image categories that are combinational attribute-object pairs, where all attributes and objects have appeared in the combination categories seen during the training phase. In the testing phase, unseen combination categories can be recognized. Transfer learning: Transfer learning is a machine learning method that can be understood as a pre-trained model being reused for another task to improve the learning efficiency of the new task. Comparison of image and text pre-training: The contrasting image-text pre-training method uses a large number of collected image-text pairs as training data, maximizing the similarity between paired image and text features while minimizing their differences from other image and text features. This approach allows for the learning of cross-modal features with good generalization performance. Model fine-tuning: Model fine-tuning is a specific transfer learning approach. Given a pre-trained model, a subset of the model's parameters are trained on a downstream dataset to make the model more applicable to the specific downstream task. Trainable parameters: Trainable parameters are those that are updated during model fine-tuning. Cue words: An input format or template designed for the downstream task that helps the pre-trained model "recall" the knowledge it "learned" during pre-training. Cue word fine-tuning: Cue word fine-tuning involves using some or all of the cue words as trainable parameters. These parameters are updated during the fine-tuning phase using data from the downstream task, thereby making the model more applicable to the specific downstream task while keeping the backbone network parameters frozen. Attention mechanism: Each input item in the model is assigned a weight, representing the degree of attention the model pays to that item. The size of the weight simulates the attentional focus of humans when processing information, effectively improving model performance. The present disclosure provides an image processing method and also relates to an image processing apparatus, another image processing method, another image processing apparatus, an image processing model training method, an image processing model training apparatus, a computing device, and a computer-readable storage medium, each of which is described in detail in the following embodiments.Referring to FIG. 1 , FIG. 1 illustrates an application scenario of an image processing method provided according to an embodiment of the present disclosure. FIG. 1 includes a client 102, an e-commerce platform server 104, and a model training platform server 106. The model training platform server 106 provides model training services to the e-commerce platform server 104. Based on a pre-trained initial image processing model and incorporating a training dataset for a target task, the model training platform server 106 can update the initial image processing model to obtain a target image processing model suitable for the target task. The target image processing model is then provided to the e-commerce platform server 104. Based on the target image processing model, the e-commerce platform server 104 can provide image processing services to users shopping on the e-commerce platform. For example, the e-commerce platform server 104 can provide users with product recognition services for products displayed in images. Based on user-provided images, the server can provide users with various attribute information about the products displayed in the images, ensuring that users have a complete understanding of the products. In specific implementations, the e-commerce platform server 104 can submit a purchase request to the model training platform server 106. Based on this purchase request, the model training platform server 106 can deliver a target image processing model that meets the requirements of the e-commerce platform server 104 to the e-commerce platform server 104. Based on this target image processing model, the e-commerce platform server 104 can provide image processing services to users. To this end, a user can send an image processing request to the e-commerce platform server 104 via the client 102. This image processing request includes the image to be processed. The e-commerce platform server 104 can input the image to be processed into the target image processing model, obtain attribute information of the target object displayed in the image to be processed, and send this attribute information to the client 102. The client 102 then displays this attribute information to the user via a display interface. Furthermore, the e-commerce platform server 104 can monitor feedback from the client 102 to maintain the target image processing model. If the feedback information indicates that the prediction result of the target image processing model is accurate but the performance is poor, the e-commerce platform server 104 can fine-tune the target image processing model. Alternatively, the e-commerce platform server 104 can send a model fine-tuning request to the model training platform server 106, and the model training platform server 106 can further update the target image processing model. This ensures the processing performance of the target image processing model.The image processing method provided in the embodiments of the present disclosure can be applied to image processing services provided by e-commerce platforms to users in the e-commerce field. This allows for further identification of attribute information of products that users wish to purchase, enabling users to fully understand product information and thus ensuring a better shopping experience. Referring to Figure 2, a flowchart of an image processing method provided in accordance with an embodiment of the present disclosure is shown, specifically comprising the following steps. Step 202: Determine an image to be processed. Specifically, the image processing method provided in the embodiments of the present disclosure can be applied to both end-side devices and cloud-side devices. The image to be processed can be understood as an image requiring processing. In image classification tasks, the image to be processed can be an image requiring classification; in object detection tasks, the image to be processed can be an image requiring object detection. It is understood that this image processing model can be applied to any image processing task, and the embodiments of the present disclosure are not limiting in this regard. Taking the e-commerce field as an example, the image processing method provided in the embodiments of the present disclosure can be provided to users of e-commerce platforms. When shopping on the e-commerce platform, users can use this image processing method to process images of the desired products to understand their attributes, such as the material, model, size, and market price. Ultimately, the attributes of the products can be obtained, allowing users to fully understand the product information and ensuring a better shopping experience. This image processing method can also be applied to other fields. For example, in digital media scenarios, this image processing method can be used to edit, review, and perform multimodal search on media content. Specifically, when a media outlet publishes a news event, the content typically includes text and images describing the news event. Using this image processing method, the published image information can be processed to obtain the corresponding text description information. By detecting whether the text information and the text description information match, the published content can be reviewed and verified, ensuring its accuracy and relevance. Step 204: Input the image to be processed into a target image processing model to obtain target text information for the image to be processed. The target image processing model is obtained by updating an initial image processing model based on an image sample determined according to the target task and the matching results of at least two text labels corresponding to the image sample. Specifically, after determining the image to be processed, the image to be processed can be input into a pre-trained target image processing model to obtain the target text information for the image to be processed.The target text information can be understood as the textual description of the image to be processed. For example, the target text information can be a textual description of the target object displayed in the image to be processed, or it can be information about the type of the target object displayed in the image to be processed. The target text information can also include attribute information of the target object in the image to be processed, such as material, model, and size. For example, if the image to be processed shows a table, the target text information can be "wooden table," where "table" is set to describe the target object displayed in the image to be processed, and "wood" is set to describe the material of the target object displayed in the image to be processed. The initial image processing model can be understood as a multimodal processing model that has been pre-trained using image and text comparison. Specifically, when pre-training the initial image processing model using image-text comparisons, the model can be pre-trained based on a large number of image-text sample pairs, ensuring that the features of the image samples and the text samples in these pairs are similar. The target task can be understood as the specific application task of the initial image processing model. For example, in the e-commerce field, the target task can be a product classification task or a product description task, while in the digital media field, the target task can be a news review task. It is understandable that different target tasks mean that the training data used when updating the initial image processing model will also be different. For example, when the target task is product classification, the training data used when updating the initial image processing model can be product images and their corresponding product categories. Furthermore, for an apparel product classification task, the training data can be images of shirts and their corresponding category "tops," images of jeans and their corresponding category "pants," images of sneakers and their corresponding category "shoes," and so on. When the target task is news review, the training data used when updating the initial image processing model can be news text and news images. The target image processing model can be understood as the image processing model obtained by updating the initial image processing model under the target task. When updating the initial image processing model, the at least two text labels corresponding to the image sample can be understood as at least two textual contents that are similar or related to the image sample. In one embodiment of the present disclosure, the text labels can be grouped as sum text labels, which can include the target object and its attribute information. For example, if the image sample is a wooden table, the corresponding text label is "wooden table," where "table" is the target object in the image sample, and "wood" is the material information (i.e., attribute information) of the target object.For example, let's say an image sample is a pair of jeans, and its corresponding text label is "jeans," where "pants" is the target object in the image sample, and "denim" is the material information of the target object. For example, if the image sample is jeans, and the target image processing model's target task is apparel classification, then during the application of the target image processing model, in order to ensure that the target image processing model outputs a more specific prediction result for the jeans image input to the target image processing model, "jeans," when updating the initial image processing model, the image sample can be jeans, and the at least two text labels corresponding to the image sample can be jeans, suit pants, sweatpants, cotton shirt, etc. The initial image processing model is updated using the matching results between the image sample and each of the at least two text labels. The update goal is to make the features of the jeans image sample similar to the features of its corresponding text label, and to make the features of the jeans image sample dissimilar to the features of other associated text labels, such as suit pants, shorts, and shirt. Updating the initial image processing model with at least two text labels can provide richer semantic information in specific task scenarios, thereby achieving zero-sample recognition. The image processing method provided by the embodiments of the present disclosure can be configured to implement combined zero-shot recognition. Zero-shot recognition can be understood as leveraging semantic information to achieve cross-modal knowledge transfer, thereby leveraging visible class information to learn and recognize previously unlearned image categories. Combined zero-shot recognition can be understood as an implementation method for zero-shot recognition, where image categories are combined attribute-object pairs (i.e., attribute information and target objects), where all attributes and objects are part of the training data. During the testing and application phases, the target image processing model can identify combined categories that are not part of the training data. For example, during training, text labels include "wooden table," "wooden chair," "plastic table," and "stone table." Because at least two text labels are input into the initial image processing model during training, providing more semantic information to the initial image processing model, even though the training text labels do not include "plastic chair," the at least two text labels provide the initial image processing model with the semantic information "plastic" and "chair." Based on this, during application, for an image to be processed showing a plastic chair, the target image processing model can identify the target text information of the image to be processed as "plastic chair."Specifically, when updating the initial image processing model, an image sample and at least two text labels corresponding to the image sample can be determined based on the target task; the image sample and the at least two text labels are input into the initial image processing model; the initial image features corresponding to the image sample and target text features corresponding to the text labels are determined in the initial image processing model; target image features corresponding to the image sample are determined based on the target text features corresponding to the text labels and the initial image features; and the initial image processing model is updated based on the matching results between the target image features and the target text features corresponding to the text labels to obtain a target image processing model. In practical applications, to adapt the image processing model to the target task, before inputting the image to be processed into the target image processing model, the initial image processing model can be updated based on the training data determined for the target task to obtain the target image processing model. Referring to FIG. 3 , FIG. 3 shows a flowchart for updating the target image processing model in an image processing method according to an embodiment of the present disclosure. The target image processing model updating steps are described in steps 302 to 310 below. Before inputting the image to be processed into the target image processing model, the following steps are further included: Step 302: Determining an image sample and at least two text labels corresponding to the image sample based on the target task. Specifically, a target training dataset can be determined based on the target task, the target training dataset comprising multiple image samples and multiple text labels. Within the target training dataset, any image sample and at least two text labels corresponding to the image sample can be determined. For example, the target task may be a product image recognition scenario in the e-commerce field, the image sample may be an image showing a plastic cup, and the at least two text labels may be "glass cup," "plastic cup," and "ceramic cup." Step 304: Inputting the image sample and the at least two text labels into the initial image processing model. Specifically, after determining the image sample and at least two text labels, the image sample and the at least two text labels can be input into a pre-trained initial image processing model. Continuing with the above example, an image of a plastic cup and at least two text labels, "glass cup," "plastic cup," and "ceramic cup," are input into the pre-trained initial image processing model. Step 306: In the initial image processing model, determine the initial image features corresponding to the image sample and determine the target text features corresponding to the text labels.Specifically, after inputting an image sample and at least two text labels into the initial image processing model, the initial image processing model can process the image sample and each text label separately to obtain initial image features and target text features corresponding to each text label. Continuing with the above example, the initial image processing model processes an image sample showing a plastic cup to determine initial image feature V0 corresponding to the image sample, target text feature T1 corresponding to the text label "glass cup," target text feature T2 corresponding to the text label "plastic cup," and target text feature T3 corresponding to the text label "ceramic cup." The initial image processing model can include an image encoder. On the image side, the image encoder can be used to encode the image sample to obtain initial image features corresponding to the image sample. In practical applications, the initial image features output by the image encoder can be global visual features of the image sample. Specifically, the image encoder can model a sequence of input image sample slices. On the text side, feature processing can be performed on each text label to obtain target text features corresponding to each text label. This can be specifically implemented as follows: Determining target text features corresponding to each text label includes: performing feature processing on each text label to obtain initial text features corresponding to each text label; determining, from the initial text features corresponding to each text label, a first initial text feature corresponding to a first text label and other initial text features corresponding to other text labels, where the first text label is any one of the text labels, and the other text labels are text labels other than the first text label; performing self-attention processing on the first initial text feature and the other initial text features to obtain a first target text feature corresponding to the first text label; and determining other target text features corresponding to the other text labels based on the processing steps of the first target text feature corresponding to the first text label. The initial image processing model may include a self-attention processing module. Specifically, feature processing can be performed on each text label to obtain initial text features corresponding to each text label. The self-attention processing module then performs self-attention processing on the first initial text feature and the other initial text features to obtain a first target text feature corresponding to the first text label. Similarly, similar processing is performed on each text label to obtain target text features corresponding to each text label.In a specific implementation, when performing self-attention processing on the first initial text feature and the other initial text features, a feature similarity between the first initial text feature and the other initial text features can be calculated. Based on this feature similarity, the first text feature and the other initial text features are fused to obtain the first target text feature. Taking the first initial text feature 1 and the other initial text features 2 and 3 as an example, a first similarity is calculated between the first initial text feature 1 and the other initial text features 2, and a second similarity is calculated between the first initial text feature 1 and the other initial text features 3. Based on this first and second similarities, the first initial text feature 1 and the other initial text features 2 and 3 are fused to enable the first initial text feature 1 to learn similar information from the other initial text features 2 and 3, thereby obtaining the first target text feature. Continuing with the above example, feature processing can be performed on the text label "glass cup" to determine the initial text feature T10 corresponding to the text label "glass cup." Feature processing can also be performed on the text label "plastic cup" to determine the initial text feature T20 corresponding to the text label "plastic cup." Feature processing can also be performed on the text label "ceramic cup" to determine the initial text feature T30 corresponding to the text label "ceramic cup." Of these three text labels, if the first text label "glass cup" is determined, then the other text labels are "plastic cup" and "ceramic cup." The first initial text feature T10, and the other initial text features T20 and T30 are used. A self-attention processing module can be used to perform self-attention processing on the first initial text feature T10 and the other initial text features T20 and T30. The goal is to enable the first initial text feature T10 to learn information from the other initial text features T20 and T30, thereby obtaining the first target text feature TL corresponding to the first text label "glass cup." Similarly, initial text features T20 and T30 are processed. A similar self-attention process is performed to obtain target text features T2 corresponding to the text label "plastic cup" and T3 corresponding to the text label "ceramic cup." In summary, the self-attention processing module performs self-attention on the initial text features corresponding to each text label to obtain target text features corresponding to each text label. This enables mutual learning between the initial text features of each text label. By promoting the transfer of relevant knowledge between the initial text features, updates are achieved based on the target task context, allowing the initial image processing model to learn more discriminative text features.In specific implementations, to enable the initial image processing model to adapt to the distribution of the training dataset for the target task, text labels can be processed by constructing prompt words. The specific implementation is as follows: Feature processing of each text label to obtain initial text features corresponding to each text label includes: converting a first text label into a first text vector; adding a parameter vector to the first text vector to obtain a first vector sequence corresponding to the first text label; determining a first initial text feature corresponding to the first text label based on the first vector sequence; and determining other initial text features corresponding to other text labels based on the processing steps for the first initial text feature corresponding to the first text label. The parameter vector can be understood as a trainable parameter vector during the update of the initial image processing model. If the first text label includes a target object and its attribute information, converting the first text label into the first text vector can be understood as splitting the first text label into the target object and its attribute information, and converting the target object and its attribute information into the first text vector. For example, if the first text label is "wooden table," the first text label is converted into a first text vector corresponding to the text vector for "table" and the text vector for "wood." In specific implementation, a preset relationship table between text and vectors can be queried to convert the first text label into a first text vector. A parameter vector is then added to the first text vector, forming a first vector sequence corresponding to the first text label from the first text vector and the parameter vector. Based on this first vector sequence, the first initial text feature corresponding to the first text label is determined. Similarly, the initial text feature corresponding to each text label can be determined. In practical applications, the parameter vector can be added as a prefix to the first text vector. Continuing with the above example, for the text label "glass cup," we can use a table lookup to convert this text label into a text vector corresponding to "glass" and a text vector corresponding to "cup." A parameter vector is then added to the text vector to form a first vector sequence: (parameter vector p1, parameter vector p2, text vector corresponding to "glass," text vector corresponding to "cup"). Based on this first vector sequence, we determine the initial text feature T10 corresponding to the text label "glass cup." Similarly, we determine the initial text feature T20 corresponding to the text label "plastic cup" and the initial text feature T30 corresponding to the text label "ceramic cup."In summary, by converting text labels into text vectors, adding parameter vectors to form a vector sequence, and determining the initial text features of the text labels based on the vector sequence, prompt word construction is achieved. Furthermore, converting combined text labels into text vectors can provide rich semantic information, facilitating the identification of combinations missing during the training phase during subsequent model application. Specifically, when determining the first initial text features based on the first vector sequence, information exchange between the vectors included in the first vector sequence can be used to achieve mutual learning between the vectors. This is specifically implemented as follows: Determining the first initial text features corresponding to the first text label based on the first vector sequence includes: performing an information exchange process on the first text vector and the parameter vector in the first vector sequence, determining a target vector in the first vector sequence based on the processing results, and using the target vector as the first initial text feature corresponding to the first text label. The target vector can be understood as a vector that can represent the global features in the first vector sequence. The target vector can be any vector in the first vector sequence after the information exchange process. Specifically, the first text vector and the parameter vector in the first vector sequence can be exchanged, and the target vector set to represent the global features in the first vector sequence can be determined based on the processing results. The target vector is used as the first initial text feature corresponding to the first text label. In practical applications, the initial image processing model may include a text encoder. The first vector sequence may be input into the text encoder, which then processes the first vector sequence and obtains the target vector output by the text encoder. Continuing with the above example, for the first vector sequence corresponding to the text label "glass cup" (parameter vector p1, parameter vector p2, text vector corresponding to "glass", text vector corresponding to "cup"), when performing information exchange processing on the vectors in the first vector sequence, taking the text vector corresponding to "glass" as an example, a preset aggregation strategy may be used to aggregate parameter vector p1, parameter vector p2, and the text vector corresponding to "cup" into the text vector corresponding to "glass", thereby obtaining the processed text vector for "glass". The preset aggregation strategy may, for example, be aggregation based on the similarity between vectors or aggregation based on the weight of the vectors. Similarly, the processed text vector for "cup", the processed parameter vector p1, and the processed parameter vector p2 are obtained. In summary, by using the text encoder to perform information exchange processing on the vectors in the first vector sequence, mutual learning between the text contents in the combined text labels can be achieved, providing rich semantic information for the subsequent image processing model to predict the combined text that was missing in the training phase. Where tj,j represents the number of attributes, j represents the number of objects, and P[j] can be understood as the first vector sequence corresponding to the text label consisting of the i-th attribute and the j-th object. For example, if the attributes include 1 wood and 2 stones, and the objects include 1 table and 2 chairs, then P1,2 represents the first vector sequence corresponding to the text label "wooden chair." S represents the attribute (i.e., the attribute information of the target object), and o represents the object (i.e., the target object). The constructed prompt word is input into the text encoder Et, and the initial text features obtained are tj,j = Et(P[j]. For the N initial text features t" of the N attribute-object pairs, these N initial text features are concatenated to obtain the initial vector matrix T composed of the N initial text features of the N attribute-object pairs. This initial vector matrix T is then input into the self-attention processing module, which outputs the target vector matrix T = softmax(%)v corresponding to the N attribute-object pairs. Based on this target vector matrix, the target vector matrix for each attribute-object pair can be obtained. The trainable parameter matrix (i.e., attention matrix) in the attention processing module consists of Q (Query), K (Key), and V (Value). It is understood that the self-attention processing module can obtain a matching score by matching Q and K, and multiply this matching score by V to aggregate relevant information. d is the dimension of the attention matrix. On the image side, for an image sample corresponding to a text label, the image sample is input into the image encoder eagle to obtain the global visual features (i.e., initial image features) X of the image sample. Step 308: Determine the target image features corresponding to the image sample based on the target text features corresponding to each text label and the initial image features. Specifically, after determining the target text features corresponding to each text label and the initial image features corresponding to the image sample, the target image features corresponding to the image sample can be determined based on the target text features and the initial image features. In a specific implementation, to achieve cross-modal de-steering of initial image features, a cross-attention processing mechanism can be used to process the initial image features and target text features, enabling the target image features to adaptively select and integrate semantically similar text features. This is specifically implemented as follows: Determining the target image features corresponding to the image sample based on the target text features corresponding to the text labels and the initial image features includes performing cross-attention processing on the target text features corresponding to the text labels and the initial image features to obtain the target image features corresponding to the image sample. The initial image processing model may include a cross-attention processing module. Specifically, cross-attention processing can be performed on the target text features corresponding to the text labels and the initial image features, and cross-modal de-steering of the initial image features using the target text features to obtain the target image features corresponding to the image sample. In a specific implementation, the similarity between the initial image features and each target text feature can be calculated. Based on the similarity, the weight of each target text feature is determined. Based on the weight of each target text feature, a weighted summation process is performed on each target text feature to obtain a target feature. The target feature is then fused with the initial image feature to obtain a target image feature.Continuing with the above example, the target text features T1 corresponding to the text label "glass cup," T2 corresponding to the text label "plastic cup," and T3 corresponding to the text label "ceramic cup" are input into the cross-attention processing module along with the initial image features V0. The cross-attention processing module outputs the target image features V. In practical applications, the aforementioned global visual features x and the updated text features (i.e., the target vector matrix composed of N target text features) are input into the cross-attention processing module to perform cross-modal correction on the visual features: x = softmax (V, Q = xW^, K = TW^, V = TW^). The visual features are X − X + λ • λ, where λ is a trainable parameter vector used to control the correction strength of each dimension. The cross-attention mechanism enables visual features to adaptively select and integrate semantically similar text features, thereby narrowing the distance between cross-modal features that should be matched, thereby achieving cross-modal correction. In summary, by performing cross-attention processing on each target text feature and the initial image feature, the target image feature is obtained. This allows the target image feature to learn information from each target text feature, enabling the target image feature to adaptively select and integrate semantically similar text features, narrowing the distance between the target image feature and the matching target text feature, thereby achieving cross-modal correction, reducing the possibility of mismatches, and further improving the processing performance of the target image processing model. Step 310: Update the initial image processing model based on the matching results between the target image feature and the target text features corresponding to the text labels to obtain a target image processing model. Specifically, after determining the target image feature and each target text feature, the initial image processing model can be updated through cross-modal matching. The matching result between the target image feature and each target text feature can be understood as the similarity between the target image feature and each target text feature. Specifically, updating the initial image processing model based on the matching result between the target image feature and each target text feature corresponding to each text label to obtain the target image processing model includes: calculating a model loss value based on the matching result between the target image feature and each target text feature corresponding to each text label; and updating the initial image processing model based on the model loss value until a target image processing model that meets a training stop condition is obtained. The model loss value can be calculated using a cross entropy loss function. The training stop condition can be understood as the number of training times reaching a preset number threshold or the model loss value reaching a preset loss value threshold. Specifically, the model loss value can be calculated based on the similarity between the target image feature and each target text feature, and the initial image processing model can be updated based on the model loss value. In practical applications, the target image feature and each target text feature can be used to calculate the recognition probability of each text label by the initial image processing model. Continuing with the above example, we can use the target image feature V and the target text feature T1 to calculate the recognition probability a of the target image processing model recognizing the text label "glass cup." Using the target image feature V and the target text feature T2, we can calculate the recognition probability b of the target image processing model recognizing the text label "plastic cup." Using the target image feature V and the target text feature T3, we can calculate the recognition probability c of the target image processing model recognizing the text label "ceramic cup."Calculate the model loss value according to the recognition probabilities a, b, and c. In practical applications, in order to reduce the number of parameters to be trained, the parameters of the text encoder and the image encoder included in the initial image processing model can be frozen, and the parameters of the self-attention processing module and the cross-attention processing module can be updated. The specific implementation method is as follows: The initial image processing model includes a self-attention processing module and a cross-attention processing module; correspondingly, updating the initial image processing model according to the model loss value until a target image processing model that meets the training stop condition is obtained includes: updating the self-attention processing module and the cross-attention processing module according to the model loss value until a target image processing model that meets the training stop condition is obtained. Specifically, the self-attention processing module includes a trainable parameter matrix, and the cross-attention processing module includes a trainable parameter matrix. When updating the self-attention processing module and the cross-attention processing module, the parameter matrix included in the self-attention processing module can be updated, and the parameter matrix included in the cross-attention processing module can be updated. In practical applications, after extracting multi-modal features at the text end and the visual end respectively, we can calculate the probability of the recognition combination through cross-modal matching as: p(cijx) = \(\frac{\exp(x - t_{ij} / T)}{\sum_{k}\exp(x - t_{ik} / T)}\), where \(t_{ij}\) refers to the combined text label.

[0002] £匚 ' exp(x - t k / T)

[0003] (i.e., the attribute-object pair) of the predictable space, \(T\in R\) is the pre-training temperature parameter of the contrastive text-image pre-training model (i.e., the initial image processing model). After obtaining the recognition probabilities of all N attribute-object pairs, the training loss (i.e., the model loss value) can be calculated using cross-entropy as \(£ = -\frac{1}{|X|}\sum_{x\in X}\log p(c|x)\), where \(X\) refers to the set of all images.

[0004] C represents an attribute-object pair. During the training phase, the gradient backpropagation of the model loss can be used to update all trainable parameters in the model. In summary, by freezing the parameters of the text encoder and image encoder and updating the parameters of the self-attention processing module and the cross-attention processing module, the number of parameters required for training can be reduced during transfer learning, thereby reducing computational complexity and parameter storage overhead, achieving parameter-efficient transfer learning. Transfer learning is a machine learning method in which a pre-trained model is reused for another task to improve the learning efficiency of that task. The method of updating a pre-trained initial image processing model to obtain a target image processing model, as provided in the embodiments of the present disclosure, is a form of transfer learning. Furthermore, to make the initial image processing model's backbone network (i.e., the text encoder and image encoder) parameters frozen while making the model more suitable for the target task, the constructed prompt word (i.e., the parameter vector added to the first text vector) can be used as a trainable parameter and updated. This is specifically implemented as follows: The parameter vector is updated based on the matching results between the target image features and the target text features corresponding to the text labels. In practical applications, after obtaining a target image processing model that meets the training stop criteria, the target image processing model can be tested. During the model testing phase, the recognition probability of each text label by the target image processing model can be used to obtain a prediction result for the image sample. The text label with the highest recognition probability among the text labels is used as the prediction result. In practical applications, the prediction result of the target image processing model for an image is: c = argmaXcjjecW(P(Ci,j|x)). Continuing with the above example, the target image processing model has a recognition probability a for the text label "glass cup," b for the text label "plastic cup," and c for the text label "ceramic cup." If b is greater than a and greater than c, then recognition probability b is the highest, and the text label "plastic cup" is the predicted result. Based on this prediction result and the image sample "plastic cup," it can be determined that the prediction result is accurate, indicating that the performance of the target image processing model meets the requirements. If the prediction result and the image sample are inconsistent, the target image processing model needs to be fine-tuned. It is understood that the test data used in the testing phase can be the same as or different from the training data used in the training phase, and this is not limited in the present embodiment. In summary, by testing the trained target image processing model, it is convenient to determine whether to continue training the target image processing model or fine-tune it based on the test results, thereby further ensuring the processing performance of the target image processing model.In summary, in the above method, the target image processing model configured to process images is obtained by updating the initial image processing model based on the image samples and text labels determined for the target task. This enables the initial image processing model to learn information about the training data for the target task, thereby ensuring the performance of the updated target image processing model in processing the target task. Furthermore, the initial image processing model is updated based on the matching results between the image samples and at least two text labels corresponding to the image samples, enabling the initial image processing model to learn information about multiple discriminative text labels. Cross-modal matching between images and text is achieved based on the matching between the image samples and the at least two text labels, thereby improving the processing performance of the multimodal processing model when applied to the target task. The image processing method provided by the present disclosure will be further described below, using the application of the image processing method in model training as an example, in conjunction with FIG4 . FIG4 shows a flow chart of the training process of an image processing method provided by one embodiment of the present disclosure, which specifically includes the following steps: Step 402: Determine an image sample and at least two text labels corresponding to the image sample based on the target task. Taking the target task of object material recognition, the image sample of a wooden table, and the text labels (i.e., combined label text) of "wooden table" and "stone table" as an example, the training of the image processing model will be described. Step 404: Input the image sample and the at least two text labels into the initial image processing model. Step 406: Determine the initial image features corresponding to the image sample using an image encoder. The initial image processing model includes an image encoder. Specifically, the image encoder can be used to determine the initial image features (i.e., visual features) V1 corresponding to the image sample "wooden table." Step 408: Use a construction module to convert the first text label into a first text vector, add a parameter vector to the first text vector, and obtain a first vector sequence corresponding to the first text label. The initial image processing model includes a construction module. The first text label is any one of the at least two text labels. Accordingly, the first vector sequence can be understood as the vector sequence corresponding to the first text label. Specifically, the construction module is used to convert the text label "wooden table" into text vector 1 corresponding to "wood" and text vector 2 corresponding to "table". Parameter vector 3 and parameter vector 4 are added as prefixes to text vector 1 and text vector 2 to obtain the vector sequence (i.e., combined prompt words) corresponding to the text label "wooden table" {3, 4, 1, 2}.Accordingly, the text label "stone table" is converted into a text vector 5 corresponding to "stone" and a text vector 2 corresponding to "table." Parameter vectors 6 and 7 are prefixed to text vectors 5 and 2, resulting in a vector sequence {6, 7, 5, 2} corresponding to the text label "stone table." Step 410: Using a text encoder, the first text vectors and the parameter vectors in the first vector sequence are exchanged. Based on the processing results, a target vector is determined in the first vector sequence and used as the first initial text feature corresponding to the first text label. The initial image processing model includes a text encoder. The first initial text feature can be understood as the initial text feature corresponding to the first text label. Specifically, using the text encoder, the vectors included in the vector sequence {3, 4, 1, 2} corresponding to the text label "wooden table" are exchanged. A target vector capable of representing global features is determined from the processed vector sequence and used as the initial text feature (i.e., combined text feature) t1 corresponding to the text label "wooden table." Accordingly, the initial text feature t2 corresponding to the text label "stone table" is obtained. Step 412: Using the self-attention processing module, self-attention processing is performed on the first initial text feature and other initial text features to obtain a first target text feature corresponding to the first text label. The initial image processing model includes the self-attention processing module. Other initial text features can be understood as initial text features corresponding to at least two text labels other than the first text label. Specifically, using the self-attention processing module, self-attention processing is performed on the initial text feature t1 corresponding to the text label "wooden table" and the initial text feature t2 corresponding to the text label "stone table" to obtain a target text feature (i.e., an updated combined text feature) T1 corresponding to the text label "wooden table" and a target text feature T2 corresponding to the text label "stone table." Step 414: Using the cross-attention processing module, cross-attention processing is performed on the target text features corresponding to the text labels and the initial image features to obtain a target image feature corresponding to the image sample. The initial image processing model includes the cross-attention processing module. Specifically, a cross-attention processing module can be used to perform cross-attention processing on the target text feature T1 corresponding to the text label "wooden table," the target text feature T2 corresponding to the text label "stone table," and the initial image feature V1 corresponding to the image sample "wooden table," to obtain the target image feature V2 corresponding to the image sample "wooden table." Step 416: Calculate the model loss value based on the matching results between the target image feature and the target text features corresponding to the text labels.Specifically, the recognition probability X of the text label "wooden table" can be calculated based on the target text feature T1 corresponding to the text label "wooden table" and the target image feature V2 corresponding to the image sample "wooden table". Based on the target text feature T2 corresponding to the text label "stone table" and the target image feature V2 corresponding to the image sample "wooden table", the recognition probability y of the text label "stone table" is calculated. Based on the recognition probability x and the recognition probability y, a model loss value is calculated. Step 418: Based on the model loss value, the self-attention processing module and the cross-attention processing module are updated until a target image processing model that meets the training stop condition is obtained. In addition, after the training of the target image processing model is completed, if it is determined that the recognition probability x is greater than the recognition probability y, the text label "wooden table" can be used as the prediction result of the target image processing model. If it is determined that the recognition probability x is less than the recognition probability y, the text label "stone table" can be used as the prediction result of the target image processing model. The target image processing model can be tested based on the prediction result to test whether the model performance meets the requirements, thereby facilitating subsequent training or fine-tuning of the model. In summary, in the above method, the target image processing model set to process the image is obtained based on the image samples and text labels determined by the target task and is updated based on the initial image processing model, thereby achieving, based on the initial image processing model, This enables the initial image processing model to learn information about the training data for the target task, thereby ensuring the performance of the updated target image processing model when processing the target task. Furthermore, the update of the initial image processing model is based on the matching results between the image sample and at least two text labels corresponding to the image sample, enabling the initial image processing model to learn information about multiple discriminative text labels. Furthermore, cross-modal matching between image and text is achieved based on the matching between the image sample and the at least two text labels, thereby improving the processing performance of the multimodal processing model when applied to the target task. The image processing method provided by the present disclosure in accordance with an embodiment of the present disclosure will be further described below, using the example of providing image processing services to users in the e-commerce field, in conjunction with FIG5 .Specifically, in the e-commerce field, e-commerce platforms can use this image processing method to provide image detection services for users who browse and purchase products on the e-commerce platform. For example, after a user sees an image of a product, although the product image generally displays information such as the product's appearance, the user may not be able to accurately obtain the product's attribute information, such as the material, size, and market price, from the image. Based on this, the e-commerce platform can provide the user with an image detection service. The service receives the image sent by the user, processes the image, and obtains the attribute information of the product in the image, thereby facilitating the user's comprehensive understanding of the product. Figure 5 shows a flowchart of the processing process of an image processing method provided according to an embodiment of the present disclosure. The specific implementation steps are as follows. Step 502: The end-side device receives the image to be detected entered by the user in the image detection request upload box, receives the user's upload instruction for the image detection request, and sends the image detection request to the cloud-side device. The end-side device displays the image detection request upload box to the user. The image detection request carries the image to be detected. The cloud-side device can be understood as an e-commerce platform server, which is configured to provide image detection services to users. Specifically, a user can select an image to be detected on the device's display interface, or capture the image using a camera. The user then clicks the "OK" control on the device's display interface. Based on the user's click instruction (i.e., upload instruction) and the image to be detected, the device generates an image detection request and uploads it to the e-commerce platform server. For example, while browsing an e-commerce platform, a user sees a picture of a notebook. To obtain information about the notebook's dimensions, material, and market price, the user uploads the picture to the image detection request upload box and clicks the "OK" control on the device's display interface. The device then generates an image detection request for the notebook image and sends it to the e-commerce platform server. Step 504: The cloud-side device receives the image detection request and determines the image to be detected. Step 506: The cloud-side device inputs the determined image to be detected into the target image processing model to obtain the target text information corresponding to the image to be detected. Specifically, the target image processing model here is a multimodal processing model updated using the aforementioned training method. Continuing with the above example, the target text information obtained by the cloud-side device may include "laptop, the size of the laptop is XX, the market price is XX, the brand is XX" and so on. Step 508: The cloud-side device sends the target text information to the end-user device. Step 510: The end-user device displays the target text information to the user in the output result display box.In addition, the end-side device can also receive the target image processing model sent by the cloud instance device and use the target image processing model to process the image to be detected on the end-side device. Corresponding to the above-mentioned method embodiment, the present disclosure also provides an embodiment of an image processing device. Figure 6 shows a schematic structural diagram of an image processing device provided by one embodiment of the present disclosure. As shown in Figure 6, the device includes: a determination module 602, configured to determine an image to be processed; an input module 604, configured to input the image to be processed into the target image processing model to obtain target text information of the image to be processed. The target image processing model is obtained by updating an initial image processing model based on an image sample determined according to the target task and the matching results of at least two text labels corresponding to the image sample. In an optional embodiment, the apparatus further includes a training module configured to: determine an image sample and at least two text labels corresponding to the image sample based on a target task; input the image sample and the at least two text labels into an initial image processing model; determine, in the initial image processing model, initial image features corresponding to the image sample and target text features corresponding to the text labels; determine target image features corresponding to the image sample based on the target text features corresponding to the text labels and the initial image features; and update the initial image processing model based on matching results between the target image features and the target text features corresponding to the text labels to obtain a target image processing model. In an optional embodiment, the training module is further configured to: perform cross-attention processing on the target text features corresponding to the text labels and the initial image features to obtain the target image features corresponding to the image sample. In an optional embodiment, the training module is further configured to: perform feature processing on each text label to obtain each initial text feature corresponding to the each text label; determine, among the initial text features corresponding to the each text label, a first initial text feature corresponding to a first text label and other initial text features corresponding to other text labels, wherein the first text label is any one of the text labels, and the other text labels are text labels among the text labels other than the first text label; perform self-attention processing on the first initial text feature and the other initial text features to obtain a first target text feature corresponding to the first text label; and determine other target text features corresponding to the other text labels according to the processing steps of the first target text feature corresponding to the first text label.In an optional embodiment, the training module is further configured to: convert the first text label into a first text vector; add a parameter vector to the first text vector to obtain a first vector sequence corresponding to the first text label; determine a first initial text feature corresponding to the first text label based on the first vector sequence; and determine other initial text features corresponding to the other text labels based on the processing steps of the first initial text feature corresponding to the first text label. In an optional embodiment, the training module is further configured to: perform information exchange processing on the first text vector and the parameter vector in the first vector sequence, determine a target vector in the first vector sequence based on the processing results; and use the target vector as the first initial text feature corresponding to the first text label. In an optional embodiment, the training module is further configured to: calculate a model loss value based on the matching results between the target image feature and the target text features corresponding to the text labels; and update the initial image processing model based on the model loss value until a target image processing model that meets the training stop condition is obtained. In an optional embodiment, the initial image processing model includes a self-attention processing module and a cross-attention processing module; the training module is further configured to update the self-attention processing module and the cross-attention processing module based on the model loss value until a target image processing model that meets the training stop condition is obtained. In an optional embodiment, the training module is further configured to update the parameter vector based on the matching results between the target image features and the target text features corresponding to the text labels. In summary, in the above-mentioned device, the target image processing model configured to process images is updated based on the image samples and text labels determined for the target task and obtained based on the initial image processing model. This enables the initial image processing model to learn information about the training data for the target task based on the initial image processing model, thereby ensuring the performance of the updated target image processing model in processing the target task. Furthermore, the initial image processing model is updated based on the matching results between the image sample and at least two text labels corresponding to the image sample. This enables the initial image processing model to learn information about multiple discriminative text labels. Furthermore, cross-modal matching between image and text is achieved based on the matching between the image sample and the at least two text labels, thereby improving the processing performance of the multimodal processing model when applied to the target task. The above is an exemplary embodiment of an image processing device according to this embodiment.It should be noted that the technical solutions of this image processing device and the technical solutions of the aforementioned image processing method share the same concept. For details not described in detail in the technical solutions of the image processing device, please refer to the description of the technical solutions of the aforementioned image processing method. Corresponding to the aforementioned method embodiments, embodiments of the present disclosure also provide an image processing method for use on a cloud-side device. FIG7 shows a flowchart of another image processing method according to one embodiment of the present disclosure, which specifically includes the following steps: Step 702: Receive an image processing request sent by a terminal-side device, wherein the image processing request carries an image to be processed; Step 704: Input the image to be processed into a target image processing model to obtain target text information for the image to be processed, wherein the target image processing model is obtained by updating an initial image processing model based on a matching result between an image sample determined according to a target task and at least two text labels corresponding to the image sample; Step 706: Send the target text information to the terminal-side device. Furthermore, the image processing method provided in embodiments of the present disclosure can also be applied to a terminal-side device. The terminal-side device can obtain a trained target image processing model from a cloud-side device and deploy the target image processing model on the terminal-side device. The end-side device receives a user click command on the end-side device's display interface, directly inputs the image to be processed into the target image processing model deployed on the end-side device, obtains target text information output by the target image processing model, and displays the target text information to the user through the display interface. In the above method, the target image processing model configured to process images is updated based on the initial image processing model, based on image samples and text labels determined for the target task. This enables the initial image processing model to learn information from the training data for the target task, thereby ensuring the performance of the updated target image processing model in processing the target task. Furthermore, the initial image processing model is updated based on the matching results between image samples and at least two text labels corresponding to the image samples, enabling the initial image processing model to learn information about multiple discriminative text labels. Cross-modal matching between images and text is achieved based on the matching between image samples and at least two text labels, thereby improving the processing performance of the multimodal processing model when applied to the target task. Corresponding to the above method embodiment, the present disclosure further provides another image processing device embodiment. FIG8 shows a schematic structural diagram of another image processing device provided by an embodiment of the present disclosure.As shown in Figure 8, the apparatus includes: a receiving module 802 configured to receive an image processing request sent by an end-side device, wherein the image processing request carries an image to be processed; an input module 804 configured to input the image to be processed into a target image processing model to obtain target text information of the image to be processed, wherein the target image processing model is obtained by updating an initial image processing model based on image samples determined for a target task and matching results of at least two text labels corresponding to the image samples; and a sending module 806 configured to send the target text information to the end-side device. In the apparatus described above, the target image processing model configured to process images is updated based on the initial image processing model, based on the image samples and text labels determined for the target task. This enables the initial image processing model to learn information from the training data for the target task, thereby ensuring the performance of the updated target image processing model in processing the target task. Furthermore, the initial image processing model is updated based on the matching results between the image sample and at least two text labels corresponding to the image sample. This enables the initial image processing model to learn information about multiple discriminative text labels. Furthermore, cross-modal matching between image and text is achieved based on the matching between the image sample and at least two text labels, thereby improving the processing performance of the multimodal processing model when applied to the target task. The above is a schematic diagram of an image processing device according to this embodiment. It should be noted that the technical solutions of this image processing device and the technical solutions of the aforementioned image processing method are based on the same concept. For details not described in detail in the technical solutions of the image processing device, please refer to the description of the technical solutions of the aforementioned image processing method. Corresponding to the above method embodiment, the disclosed embodiment also provides an image processing model training method, which specifically includes the following steps. An image sample and at least two text labels corresponding to the image sample are determined based on the target task; the image sample and the at least two text labels are input into an initial image processing model; in the initial image processing model, initial image features corresponding to the image sample are determined, and target text features corresponding to the text labels are determined; the initial image features are processed based on the target text features corresponding to the text labels to obtain target image features corresponding to the image sample; and based on the matching results between the target image features and the target text features corresponding to the text labels, the initial image processing model is updated to obtain a target image processing model. Specifically, the training of the image processing model is the same as the updating process of the target image processing model in the image processing method shown in FIG. 3 , and will not be repeated here.It is understood that image processing model training can be performed on either a cloud-side device or an end-side device. When this image processing model training method is applied to a cloud-side device, after obtaining the target image processing model, the model parameters of the target image processing model can also be sent to the end-side device. In the above method, the target image processing model used to process images is updated based on the image samples and text labels determined for the target task and obtained based on the initial image processing model. This enables the initial image processing model to learn information from the training data for the target task, thereby ensuring the performance of the updated target image processing model in processing the target task. Furthermore, the update of the initial image processing model is based on the matching results between the image samples and at least two text labels corresponding to the image samples. This enables the initial image processing model to learn information about multiple discriminative text labels. Cross-modal matching between images and text is achieved based on the matching between the image samples and the at least two text labels, thereby improving the processing performance of the multimodal processing model when applied to the target task. Corresponding to the above-mentioned method embodiments, the present disclosure also provides an embodiment of an image processing model training device. FIG9 shows a schematic structural diagram of an image processing model training device provided in one embodiment of the present disclosure. As shown in FIG9 , the device includes: a first determination module 902, configured to determine an image sample and at least two text labels corresponding to the image sample based on a target task; an input module 904, configured to input the image sample and the at least two text labels into an initial image processing model; a second determination module 906, configured to determine, in the initial image processing model, initial image features corresponding to the image sample and target text features corresponding to the text labels; a processing module 908, configured to process the initial image features based on the target text features corresponding to the text labels to obtain target image features corresponding to the image sample; and an updating module 910, configured to update the initial image processing model based on matching results between the target image features and the target text features corresponding to the text labels to obtain a target image processing model. In summary, in the above-mentioned apparatus, the target image processing model configured to process images is obtained by updating the initial image processing model based on the image samples and text labels determined for the target task. This enables the initial image processing model to learn information from the training data for the target task, thereby ensuring the performance of the updated target image processing model in processing the target task.Furthermore, the initial image processing model is updated based on the matching results between the image sample and at least two text labels corresponding to the image sample. This enables the initial image processing model to learn information about multiple discriminative text labels and achieves cross-modal matching between image and text based on the matching between the image sample and the at least two text labels, thereby improving the processing performance of the multimodal processing model when applied to the target task. The above is a schematic diagram of an image processing model training device according to this embodiment. It should be noted that the technical solution of the image processing model training device and the technical solution of the aforementioned image processing model training method are based on the same concept. For details not described in detail in the technical solution of the image processing model training device, please refer to the description of the technical solution of the aforementioned image processing model training method. Figure 10 shows a block diagram of a computing device 1000 provided according to one embodiment of the present disclosure. Components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030. A database 1050 is configured to store data. Computing device 1000 also includes an access device 1040 that enables computing device 1000 to communicate via one or more networks 1060. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. Access device 1040 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.In one embodiment of the present application, the aforementioned components of computing device 1000, as well as other components not shown in FIG. 10 , may also be connected to one another, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 10 is for illustrative purposes only and does not limit the scope of the present application. Those skilled in the art may add or replace other components as needed. Computing device 1000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1000 may also be a mobile or stationary server. Processor 1020 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described method. The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the aforementioned method are based on the same concept. Any details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the aforementioned method. An embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these computer-executable instructions implement the steps of the aforementioned method. The above is an exemplary embodiment of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the aforementioned method are based on the same concept. Any details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the aforementioned method. An embodiment of the present disclosure also provides a computer program. When executed on a computer, the computer executes the steps of the aforementioned method. The above is an exemplary embodiment of a computer program of this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the aforementioned method are based on the same concept. Any details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the aforementioned method. The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results.In certain implementations, multitasking and parallel processing are also possible or may be advantageous. The computer instructions include computer program code, which may be in source code form, object code form, executable files, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, a USB flash drive, a removable hard drive, a magnetic disk, an optical disk, computer memory, read-only memory (ROM), random access memory (RAM), an electric carrier signal, a telecommunications signal, and software distribution media. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunications signals. It should be noted that for ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present disclosure are not limited by the order of the actions described, as certain steps may be performed in other orders or simultaneously according to the embodiments of the present disclosure. Secondly, those skilled in the art should also be aware that the embodiments described in this disclosure are preferred embodiments, and the actions and modules described are not necessarily required for the embodiments of this disclosure. In the above embodiments, the description of each embodiment has its own focus. For portions not described in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. The preferred embodiments of this disclosure disclosed above are merely provided to illustrate this disclosure. The alternative embodiments do not describe all details in detail, nor do they limit the invention to the specific implementations described. Obviously, many modifications and variations are possible based on the content of the embodiments of this disclosure. This disclosure selects and describes these embodiments in detail to better explain the principles and practical applications of the embodiments of this disclosure, thereby enabling those skilled in the art to better understand and utilize this disclosure. This disclosure is subject to the full scope of the claims and their equivalents. Industrial Applicability The image processing method and apparatus provided in the embodiments of the present disclosure are configured as a target image processing model for processing images. The target image processing model is obtained by updating the initial image processing model based on the image samples and text labels determined for the target task. This enables the initial image processing model to learn information about the training data in the target task, thereby ensuring the performance of the updated target image processing model in processing the target task.Moreover, the updating of the initial image processing model is achieved based on the matching results between the image sample and at least two text labels corresponding to the image sample, which enables the initial image processing model to learn information about multiple discriminative text labels and realize cross-modal matching between image and text based on the matching between the image sample and at least two text labels, thereby improving the processing performance of the multimodal processing model when applied to the target task.

Claims

Claims 1. An image processing method, comprising: Determine the image to be processed; The image to be processed is input into a target image processing model to obtain target text information of the image to be processed, wherein the target image processing model is obtained by updating an initial image processing model based on an image sample determined according to a target task and a matching result of at least two text labels corresponding to the image sample.

2. The method according to claim 1, before inputting the image to be processed into the target image processing model, further comprising: Determine an image sample and at least two text labels corresponding to the image sample according to a target task; The image sample and the at least two text labels are input into an initial image processing model; in the initial image processing model, initial image features corresponding to the image sample are determined, and target text features corresponding to the text labels are determined; target image features corresponding to the image sample are determined according to target text features corresponding to the text labels and the initial image features; and the initial image processing model is updated according to a matching result between the target image features and target text features corresponding to the text labels to obtain a target image processing model.

3. The method according to claim 2, wherein determining the target image feature corresponding to the image sample according to the target text features corresponding to the text labels and the initial image feature comprises: Cross-attention processing is performed on each target text feature corresponding to each text label and the initial image feature to obtain a target image feature corresponding to the image sample.

4. The method according to claim 2, wherein determining each target text feature corresponding to each text label comprises: Perform feature processing on each text label to obtain each initial text feature corresponding to the text label; determine a first initial text feature corresponding to a first text label and other initial text features corresponding to other text labels among the initial text features corresponding to the text labels, wherein the first text label is any one of the text labels, and the other text labels are text labels among the text labels except the first text label; perform self-attention processing on the first initial text feature and the other initial text features to obtain a first target text feature corresponding to the first text label; and determine other target text features corresponding to the other text labels according to the processing steps of the first target text feature corresponding to the first text label.

5. The method according to claim 4, wherein the step of performing feature processing on each text label to obtain each initial text feature corresponding to each text label comprises: Convert the first text label into a first text vector; Adding a parameter vector to the first text vector to obtain a first vector sequence corresponding to the first text label; determining a first initial text feature corresponding to the first text label according to the first vector sequence; According to the processing step of the first initial text feature corresponding to the first text label, other initial text features corresponding to the other text labels are determined.

6. The method according to claim 5, wherein determining the first initial text feature corresponding to the first text label according to the first vector sequence comprises: performing information exchange processing on the first text vector and the parameter vector in the first vector sequence, and determining a target vector in the first vector sequence according to the processing result; The target vector is used as a first initial text feature corresponding to the first text label.

7. The method according to claim 2, wherein updating the initial image processing model according to the matching result between the target image feature and each target text feature corresponding to each text label to obtain the target image processing model comprises: Calculating a model loss value according to a matching result between the target image feature and each target text feature corresponding to each text label; The initial image processing model is updated according to the model loss value until a target image processing model that meets the training stop condition is obtained.

8. The method according to claim 7, wherein the initial image processing model comprises a self-attention processing module and a cross-attention processing module; accordingly, updating the initial image processing model according to the model loss value until a target image processing model that satisfies a training stop condition is obtained comprises: According to the model loss value, the self-attention processing module and the cross-attention processing module are updated until a target image processing model that meets the training stop condition is obtained.

9. The method according to claim 5, further comprising: The parameter vector is updated according to the matching result between the target image feature and each target text feature corresponding to each text label.

10. An image processing method, applied to a cloud-side device, comprising: An image processing request is received from an end-side device, wherein the image processing request carries an image to be processed; the image to be processed is input into a target image processing model to obtain target text information of the image to be processed, wherein the target image processing model is obtained by updating an initial image processing model based on an image sample determined according to a target task and a matching result of at least two text labels corresponding to the image sample; and the target text information is sent to the end-side device.

11. A method for training an image processing model, comprising: Determine an image sample and at least two text labels corresponding to the image sample according to a target task; The image sample and the at least two text labels are input into an initial image processing model; in the initial image processing model, initial image features corresponding to the image sample are determined, and target text features corresponding to the text labels are determined; according to the target text features corresponding to the text labels, the initial image features are processed to obtain target image features corresponding to the image sample; according to the matching results between the target image features and the target text features corresponding to the text labels, the initial image processing model is updated to obtain a target image processing model.

12. An image processing device, comprising: A determination module, configured to determine an image to be processed; The input module is configured to input the image to be processed into a target image processing model to obtain target text information of the image to be processed, wherein the target image processing model is obtained by updating an initial image processing model based on an image sample determined according to a target task and a matching result of at least two text labels corresponding to the image sample.

13. A computing device, comprising: Memory and processor; The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Citation Information

Patent Citations

  • Method and device for generating description information of multimedia data, equipment and medium

    CN111723937A

  • Neural network training and classification method and device, equipment and storage medium

    CN113920406A

  • Training method and training device for image description model

    CN114090815A

  • Graph-text retrieval method based on hierarchical alignment and generalized pooling graph attention mechanism

    CN114896438A

Cited By

  • Combined zero sample image classification method based on hierarchical feature fusion

    CN119478551A

  • Cabin wallpaper generation method and device, electronic equipment and storage medium

    CN119741191A

  • Picture generation model depolarization method and system based on enhanced cue word learning

    CN119963675A