Image ordered classification model training method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an image ordered classification model training method and device, electronic equipment and storage medium. The method comprises: obtaining a plurality of types of training samples, each training sample containing a text description and a first image corresponding to the text description; performing word segmentation processing on the text description to obtain the word segmentation of each text description; performing first sorting processing on the first word in each text description, and sorting each text description based on the result of the first sorting processing; wherein the first word is a word in each text description used to identify the type of the training sample; performing second sorting processing on the second word in each text description; wherein the second word is a word in each text description other than the first word; and training an image ordered classification model based on the text description after the first sorting processing and the second sorting processing and the first image. The present disclosure can solve the overfitting problem in the related art.

Description

Technical Field

[0001] This disclosure belongs to the field of image classification technology, and more specifically, relates to an image ordered classification model training method and apparatus, electronic device, and storage medium. Background Technology

[0002] Ordered classification aims to predict features with natural or implicit sequential relationships and can be seen as a special case of ordered regression after feature discretization. Compared to traditional classification, ordered classification requires additional consideration of the ordinal attribute of features. However, related technologies suffer from overfitting due to simply training the model on the training data, meaning the trained model's performance degrades in real-world applications. Summary of the Invention

[0003] The purpose of this disclosure is to provide a method, apparatus, electronic device, and storage medium for training an image ordered classification model, in order to solve the overfitting phenomenon in related technologies, i.e., the problem of performance degradation of the trained model in practical applications.

[0004] A first aspect of this disclosure provides a method for training an image ordered classification model, comprising: Acquire multiple types of training samples, each training sample containing a text description and a first image corresponding to the text description; The text description is segmented to obtain the word segments for each text description; The first segmentation in each text description is subjected to a first sorting process, and the text descriptions are sorted based on the result of the first sorting process; wherein, the first segmentation is the segmentation in each text description used to identify the type of training sample; A second sorting process is performed on the second segment of each text description; wherein, the second segment is the segment of each text description other than the first segment; An image sorting model is trained based on the text descriptions after the first and second sorting processes and the first image.

[0005] A second aspect of this disclosure provides an image ordered classification model training apparatus, comprising: The data acquisition module is used to acquire multiple types of training samples, each training sample containing a text description and a first image corresponding to the text description; The data processing module is used to perform word segmentation on the text description to obtain word segments for each text description; to perform a first sorting process on the first word segments in each text description, and to sort each text description based on the result of the first sorting process; wherein, the first word segment is the word segment used to identify the training sample type in each text description; and to perform a second sorting process on the second word segments in each text description; wherein, the second word segment is the word segment in each text description other than the first word segment. The model training module is used to train an image ordered classification model based on the text descriptions after the first and second sorting processes and the first image.

[0006] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described image ordered classification model training method.

[0007] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described image ordered classification model training method.

[0008] The beneficial effects of the image ordered classification model training method, apparatus, electronic device, and storage medium provided in this disclosure are as follows: Unlike related technologies, the embodiments of this disclosure perform a first sorting process on the first word segment in each text description and a second sorting process on the second word segment in each text description. By sorting different types of word segments in different ways, the advantages of prior information in the text and the ordered information contained in the text description can be fully utilized, thereby improving the performance of the image ordered classification model, enhancing the semantic alignment and sorting alignment between the text description and the first image, and solving the overfitting phenomenon in related technologies, i.e., the problem of performance degradation of the trained model in practical applications. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic flowchart of an image ordered classification model training method provided in an embodiment of the present disclosure; Figure 2A schematic flowchart illustrating an image ordered classification model training method provided in a specific embodiment of this disclosure; Figure 3 This is a structural block diagram of an image ordered classification model training device provided in an embodiment of the present disclosure; Figure 4 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Detailed Implementation

[0011] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0012] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0013] Visual language pre-training (VLP) has significantly improved the performance of many downstream tasks, including segmentation, object detection, image retrieval, generation tasks, and sequence regression, through text and image matching. One related technique embeds images and text into the same representation space using contrastive loss and two independent encoders. This approach demonstrates impressive "zero-shot" performance, showcasing the powerful role of language priors. Inspired by recent advances in Natural Language Processing (NLP), another related technique focuses on improving model capabilities through prompts and adapter tuning. For example, a lightweight module is added on top of the image and text encoders to learn contextual cues for image classification. However, both approaches suffer from performance degradation in sequence classification due to the lack of sequence attributes.

[0014] To address the aforementioned issues, this embodiment considers the significant potential of language priors and proposes integrating learning sorting techniques into an image ordered classification model for sequence classification. Specifically, it proposes a training method for an image ordered classification model, which is applied to classifying images according to task types, such as age estimation, aesthetic estimation, and date estimation.

[0015] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an image ordered classification model training method according to an embodiment of the present disclosure. The method includes: S101: Obtain training samples of multiple types, each training sample containing a text description and a first image corresponding to the text description.

[0016] In this embodiment, there is a one-to-one correspondence between the text description and the first image. Multiple types of training samples are included, but are not limited to, age estimation, aesthetic estimation, and date estimation, with each type containing multiple training samples. For example, the training samples for age estimation include a text description (a photo of x years old face, where x is the age category) and the corresponding image; the training samples for aesthetic estimation include a text description (ay photo, where y is the aesthetic category) and the corresponding image; and the training samples for date estimation include a text description (a photo taken in the z region, where z is the date category) and the corresponding image.

[0017] S102: Perform word segmentation on the text description to obtain the word segments for each text description.

[0018] In this embodiment, word segmentation is the first step in text description preprocessing. Since text descriptions cannot be directly fed into the model for analysis, word segmentation breaks them down into data structures based on words, facilitating subsequent processing and analysis. For example, word segmentation of the text description "a photo of 1 years oldface" yields "a", "photo", "1", "years", "old", and "face".

[0019] S103: Perform a first sorting process on the first segmentation in each text description, and sort each text description based on the result of the first sorting process; wherein, the first segmentation is the segmentation in each text description used to identify the type of training sample.

[0020] In this embodiment, the first word segmentation is the word segmentation used to identify the type of training sample in each text description. For example, the word "1" in the text description "a photo of 1 years old face" is used to identify that this training sample is an age estimation type training sample. The first sorting process refers to sorting the first word segmentation according to its level. For example, the text description of the age estimation type training sample (a photo of (x) years old face) is sorted according to the size of the word segmentation x; the text description of the aesthetic estimation type training sample (a (y) photo) is sorted according to the level of the word segmentation y, such as "unacceptable", "flawed", "ordinary", "professional", "exceptional"; the text description of the age estimation type training sample (a photo taken in the (z)) is sorted according to the level of the word segmentation z, such as "1930s", "1940s", ..., "1990s".

[0021] This embodiment sorts the text descriptions based on the results of the first sorting process, which can help the image ordered classification model make full use of the ordered information contained in the text information. This is beneficial to solving the overfitting phenomenon in related technologies, that is, the problem of performance degradation of the trained model in practical applications.

[0022] S104: Perform a second sorting process on the second segment of each text description; wherein, the second segment is the segment of each text description other than the first segment.

[0023] In this embodiment, the second word segmentation refers to the word segmentation in each text description excluding the first word segmentation. For example, in the text description "a photo of 1 years old face", the word segmentation is "a", "photo", "years", "old", and "face". The second sorting process refers to sorting the second word segmentation in the text description according to the text description template of each type. For example, the text description template for the age estimation type training sample is "a photo of x years old face", and the second word segmentation "a", "years", "photo", "face", and "old" is sorted as "a", "photo", "years", "old", and "face". The text description after the second sorting process can fully utilize the potential of language, which helps to tailor the model's behavior to specific tasks and domains, making the model more efficient.

[0024] S105: Train an ordered image classification model based on the text descriptions after the first and second sorting processes and the first image.

[0025] The first image corresponds to the text description, and the trained image sorting model can be used to obtain the sorted classification result of the input image based on the image input to the model.

[0026] Compared to traditional classification, ordered classification requires additional consideration of the ordinal attributes of features. In related technologies, simply training a model on training data often leads to overfitting, meaning the trained model's performance degrades in practical applications. To address this issue, this embodiment trains an image ordered classification model based on text descriptions after first and second sorting processes, along with a first image. This fully leverages the advantages of prior text information and the ordered information inherent in the text descriptions, making the model more efficient and improving the performance of the image ordered classification model.

[0027] As can be seen from the above, the embodiments of this disclosure, by performing a first sorting process on the first word segment in each text description and a second sorting process on the second word segment in each text description, can give full play to the advantages of the prior information of the text and the ordered information contained in the text description, improve the performance of the image ordered classification model, enhance the semantic alignment and sorting alignment between the text description and the first image, and solve the overfitting phenomenon in related technologies, that is, the problem of performance degradation of the trained model in practical applications.

[0028] In one embodiment of this disclosure, a first sorting process is performed on the first words in each text description, including: The first word segment of each text description is input into a pre-trained attention model to obtain the ranking result.

[0029] The attention model is as follows:

[0030] Where R represents the first segmentation word, This represents the first word segment after the first sorting process. This represents the preset residual ratio. It is a fully connected function. For multi-head attention functions, This is the layer normalization function.

[0031] In this embodiment, the fully connected function in the model performs a linear transformation on the input data, mapping low-level features to high-level feature representations. The mathematical expression is: =a*x+b. The layer normalization function can solve the problems of gradient explosion and gradient vanishing in the model; its mathematical expression is: Multi-head attention functions can extend a model's ability to focus on different locations; the mathematical expression is:

[0032] =

[0033]

[0034] in, Representative value matrix, Represents dimension.

[0035] In this embodiment, the pre-trained attention model can be the RankFormer model. The token-based attention layer in the RankFormer model is used to sort the first word in each text description, thereby enhancing the sorting relationship of the text description.

[0036] In one embodiment of this disclosure, sorting the various text descriptions based on the result of a first sorting process includes: The order of the first word segment corresponding to each text description is determined as the order of each text description.

[0037] In this embodiment, the order of the first segment corresponding to each text description is determined as the order of the text descriptions themselves. That is, the text descriptions are sorted according to the level of the first segment. For example, the three text descriptions "a photo of 1 year old face", "a photo of 5 years old face", and "a photo of 2 years old face" are sorted according to the size of "1", "5", and "2", resulting in "a photo of 1 year old face", "a photo of 2 years old face", and "a photo of 5 years old face". Furthermore, the first image corresponding to each text description undergoes the same sorting adjustment as the corresponding text description to facilitate subsequent model training.

[0038] In one embodiment of this disclosure, the loss function corresponding to the attention model is: A = a1*k1 + a2*(1-k1) Where a1 is the asymmetric contrastive loss function, a2 is the cross-modal ordered pairing loss function, and k1 is the weight corresponding to the asymmetric contrastive loss function.

[0039] The asymmetric contrastive loss function is:

[0040] in, For temperature parameters, , Image features representing the first image, represent The transpose of the matrix, The image features representing the j-th first image, represent The transpose of the matrix, Represents the actual label value. B represents the text features with true label values, B is the number of the first images input into the ordered learning model in each batch, and the true label value refers to the predicted label value, which is used to observe whether the model predicts correctly.

[0041] The cross-modal ordered pairing loss function is:

[0042]

[0043] in, These are hyperparameters for the computation process. The text features described for the j-th text. Represents the actual label value. For image features with true label values, This represents a label-based distance-aware function.

[0044] In this embodiment, the original contrastive loss function is:

[0045] in, Here are the hyperparameters for the computation process: N is the total number of samples in the training set, and K is the total number of samples of a certain type in the training set. Representing image features, represent The transpose of the matrix, Represents the actual label value. Text features that represent actual label values. This represents the probability that the i-th image belongs to a certain type.

[0046] Because a batch contains many-to-many image-text mappings—in other words, different images in a batch may have the same rank, and the same rank may have different descriptions—this embodiment uses... Replace with Generate an asymmetric contrastive loss function to replace the original contrastive loss function.

[0047] Furthermore, this embodiment introduces text priors into the loss function and proposes a label-based distance-aware function, namely a cross-modal ordered pairing loss function. This fully applies language prior knowledge and distance awareness to the pairing cross loss to constrain the order relationship of images of different ranking categories in the cross-modal space.

[0048] As can be seen from the above analysis, the loss function corresponding to the attention model is the weighted result of the asymmetric contrast loss function and the cross-modal ordered pairing loss function. It fully applies language prior knowledge and distance awareness to the pairing cross loss to optimize the attention model and keep the semantic alignment and order alignment of the image and text description.

[0049] In one embodiment of this disclosure, an image ordered classification model is trained based on the text descriptions after the first and second sorting processes and the first image, including: The first prompt vector is generated based on the text descriptions after the first and second sorting processes.

[0050] An ordered image classification model is trained based on the first cue vector and the first image.

[0051] The process of generating a first prompt vector based on the text descriptions after the first and second sorting processes is known as text vectorization. This involves representing the text in vector or matrix form, which can also be understood as numerical processing of the text to facilitate the training of image sorting classification models. There are many methods for text vectorization, such as vectorized retrieval based on contrastive learning and differentiable quantization, StruBERT-based structured information retrieval, and Prompt-based document retrieval models.

[0052] In one embodiment of this disclosure, the image ordered classification model training method further includes: Randomly initialize the global context hint vector.

[0053] The first hint vector is concatenated with the global context hint vector to obtain the second hint vector.

[0054] The image sorting model trained based on the first cue vector and the first image includes: An image sorting model is trained based on the second cue vector and the first image.

[0055] In this embodiment, concatenating the first cue vector with the global context cue vector allows the image ordered classification model to better align with training samples of different types. For example, when the task type of the image ordered classification model is age estimation, the global context cue vector will indicate to the image ordered classification model that it is currently being tested on training samples of the age estimation type, thereby improving the performance of the image ordered classification model.

[0056] In one embodiment of this disclosure, the loss function corresponding to the image ordered classification model is: B = b1*k2 + b2*(1-k2) Where b1 is the cross-entropy loss function, b2 is the cross-modal ordered pairing loss function, and k2 is the weight corresponding to the cross-entropy loss function.

[0057] In this embodiment, the cross-entropy loss function can improve the accuracy of model predictions. The expression for the cross-entropy loss function is as follows:

[0058] The loss function for constructing an ordered image classification model is:

[0059]

[0060] in, For hyperparameters, Image features representing the first image, represent The transpose of the matrix, The text features representing the description of the j-th text. represent The transpose of the matrix, Represents the actual label value. The text features represent the true label values, and N is the number of the first images input into the ordered learning model in each batch.

[0061] The loss function is simplified to obtain the loss function of the image ordered classification model in the actual calculation process. Finally, the loss function of the image ordered classification model is determined to be the weighted result of the cross-entropy loss function and the cross-modal ordered pairing loss function.

[0062] In one embodiment of this disclosure, the second sorting process for the second word segment in each text description can be performed using Prompt Tuning, a technique that modifies input prompts to achieve better modeling results. Prompt Tuning is further divided into hard prompt tuning, which directly alters the non-differentiable discrete input tokens, and soft prompt tuning, which connects the embeddings of the input tokens to a trainable tensor that can be optimized through backpropagation to improve the modeling performance of the target task.

[0063] In one embodiment of this disclosure, the image ordering classification model can be a CLIP (Contrastive Language-Image Pre-training) network model. Considering the great potential of language priors, it is proposed to integrate learning ordering techniques into the CLIP network model for sequence classification.

[0064] refer to Figure 2 , Figure 2 This is a flowchart illustrating an image ordered classification model training method provided in a specific embodiment of this disclosure.

[0065] Image ordered classification model training methods include: Get the text description and its corresponding image.

[0066] The text description is input into the rankformer network model, the first word segment is sorted in the first sorting process, and the second word segment is sorted in the second sorting process to obtain the first prompt vector.

[0067] Randomly initialize the global context hint vector.

[0068] The first hint vector is concatenated with the global context hint vector to obtain the second hint vector.

[0069] The second cue vector and the image are fed into the CLIP network model for training error calculation.

[0070] Optimize image sorting classification model based on training error.

[0071] The process of feeding the second cue vector and the image into the CLIP network model for training error calculation includes: using a text encoder to extract features from the second cue vector as text features, using an image encoder to extract features from the image as image features, and calculating the training error between the text features and the image features.

[0072] In this embodiment, the image ordered classification model is a network model based on the CLIP network model with the addition of a rankformer module. The rankformer module can sort the word segments of the text description based on token attention. For example, if the text description is "a photo of 1 years old face" and "a photo of 2 years old face", the rankformer module first sorts the first word segments "1" and "2" in order of size, and then sorts the text descriptions "a photo of 1 years old face" and "a photo of 2 years old face" according to the sorting results of the word segments "1" and "2". Then, it sorts the second word segments "a", "photo", "old", "years", and "face" in the text description into "a", "photo", "years", "old", and "face". That is, in Figure 2 R in i R k R j There are three text descriptions, ordered by age, for example, R i For "a photo of 1 years old face", R k for "a photo of 5 years old face" R j for "a photo of 10 years old face". Figure 2 T in i With T j For the second word in the text description, perform a second sorting process on the second word, for example, sort "old" and "years" into "years" and "old".

[0073] In one embodiment of this disclosure, sequence classification is a unique case of image classification, where labels have sequence attributes. Mathematically, let... Indicates the first One input instance, , Represents the ordered arrangement of the true label values, where , Indicates the category of network prediction. Indicates the total number of instances. Indicates the level number, This represents the ranking relationship between different levels. Similar to ordinary classification, sequence classification aims to encode images as features. ,in It's an encoder, then a classifier is used. Calculate probability To restore Predicted labels It has the highest probability The result. The classification probability can be calculated using the following formula:

[0074]

[0075] To leverage sequence information in language, sequence classification can be transformed into a visual-language alignment task. This embodiment uses a pre-trained CLIP image feature extractor. Extracting features from the input image Construct a hard ranking template for a given sequence classification task. We obtain text features. For each template, we convert it into fixed-length tokens and then map them into 512-dimensional word embeddings. Language feature extractor. Encode the embeddings into classifier weights. This process can be represented as: Finally, we can use the above formula to calculate the ranking. The predicted probability.

[0076] Corresponding to the image ordered classification model training method in the above embodiment, Figure 3 This is a structural block diagram of an image ordered classification model training apparatus provided according to an embodiment of the present disclosure. For ease of explanation, only the parts relevant to the embodiment of the present disclosure are shown. References Figure 3 The image ordered classification model training device 20 includes: Data acquisition module 201 is used to acquire multiple types of training samples, each training sample containing a text description and a first image corresponding to the text description; The data processing module 202 is used to perform word segmentation on the text description to obtain word segments for each text description; to perform a first sorting process on the first word segments in each text description, and to sort each text description based on the result of the first sorting process; wherein, the first word segment is the word segment in each text description used to identify the type of training sample; and to perform a second sorting process on the second word segments in each text description; wherein, the second word segment is the word segment in each text description other than the first word segment. The model training module 203 is used to train an image ordered classification model based on the text descriptions after the first and second sorting processes and the first image.

[0077] In one embodiment of this disclosure, the data processing module 202 is specifically used to input the first word segmentation in each text description into a pre-trained attention model to obtain the ranking result.

[0078] The attention model is as follows:

[0079] Where R represents the first segmentation word, This represents the first word segment after the first sorting process. This represents the preset residual ratio. It is a fully connected function. For multi-head attention functions, This is the layer normalization function.

[0080] In one embodiment of this disclosure, the data processing module 202 is specifically used for: The order of the first word segment corresponding to each text description is determined as the order of each text description.

[0081] In one embodiment of this disclosure, the loss function corresponding to the attention model is: A = a1*k1 + a2*(1-k1) Where a1 is the asymmetric contrastive loss function, a2 is the cross-modal ordered pairing loss function, and k1 is the weight corresponding to the asymmetric contrastive loss function.

[0082] The asymmetric contrastive loss function is:

[0083] in, For temperature parameters, , Image features representing the first image, represent The transpose of the matrix, The image features representing the j-th first image, represent The transpose of the matrix, Represents the actual label value. B represents the text features with true label values, and B is the number of the first images input into the ordered learning model in each batch.

[0084] The cross-modal ordered pairing loss function is:

[0085]

[0086] in, These are hyperparameters for the computation process. The text features described for the j-th text. Represents the actual label value. For image features with true label values, This represents a label-based distance-aware function.

[0087] In one embodiment of this disclosure, the data processing module 202 is specifically used for: The first prompt vector is generated based on the text descriptions after the first and second sorting processes.

[0088] An ordered image classification model is trained based on the first cue vector and the first image.

[0089] In one embodiment of this disclosure, the data processing module 202 is further configured to: Randomly initialize the global context hint vector.

[0090] The first hint vector is concatenated with the global context hint vector to obtain the second hint vector.

[0091] The image sorting model trained based on the first cue vector and the first image includes: An image sorting model is trained based on the second cue vector and the first image.

[0092] In one embodiment of this disclosure, the loss function corresponding to the image ordered classification model is: B = b1*k2 + b2*(1-k2) Where b1 is the cross-entropy loss function, b2 is the cross-modal ordered pairing loss function, and k2 is the weight corresponding to the cross-entropy loss function.

[0093] See Figure 4 , Figure 4 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Figure 4 The terminal 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules / units in the above-described device embodiments, such as... Figure 3 The functions of modules 201 to 203 are shown.

[0094] It should be understood that, in the embodiments of this disclosure, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0095] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0096] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0097] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this disclosure can execute the implementation methods described in the first and second embodiments of the image ordered classification model training method provided in the embodiments of this disclosure, or they can execute the implementation methods of the terminal described in the embodiments of this disclosure, which will not be repeated here.

[0098] In another embodiment of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. The computer program can also instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0099] The computer-readable storage medium can be an internal storage unit of the terminal in any of the foregoing embodiments, such as the terminal's hard disk or memory. The computer-readable storage medium can also be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the terminal. The computer-readable storage medium is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0100] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0101] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the terminals and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed terminals and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this disclosure, depending on actual needs.

[0104] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this disclosure, and these modifications or substitutions should all be covered within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for training an image ordered classification model, characterized in that, include: Acquire multiple types of training samples, each training sample containing a text description and a first image corresponding to the text description; The text description is segmented to obtain the word segments for each text description; The first segmentation in each text description is subjected to a first sorting process, and the text descriptions are sorted based on the result of the first sorting process; wherein, the first segmentation is the segmentation in each text description used to identify the type of training sample; A second sorting process is performed on the second segment of each text description; wherein, the second segment is the segment of each text description other than the first segment; A sorted image classification model is trained based on the text descriptions after the first and second sorting processes and the first image. The method for training an ordered image classification model based on the text descriptions after the first and second sorting processes and the first image includes: A first prompt vector is generated based on the text descriptions after the first and second sorting processes. The image sorting model is trained based on the first cue vector and the first image; The image ordered classification model training method also includes: Randomly initialize the global context hint vector; The first prompt vector is concatenated with the global context prompt vector to obtain the second prompt vector; The step of training the image ordered classification model based on the first cue vector and the first image includes: An image sorting model is trained based on the second cue vector and the first image.

2. The image ordered classification model training method as described in claim 1, characterized in that, The first segmentation of each text description is sorted, including: The first word segment of each text description is input into a pre-trained attention model to obtain the ranking result; The attention model is as follows: Where R represents the first word segmentation. This represents the first word segment after the first sorting process. This represents the preset residual ratio. It is a fully connected function. For multi-head attention functions, This is the layer normalization function.

3. The image ordered classification model training method as described in claim 1, characterized in that, The sorting of each text description based on the result of the first sorting process includes: The order of the first word segment corresponding to each text description is determined as the order of each text description.

4. The image ordered classification model training method as described in claim 2, characterized in that, The loss function corresponding to the attention model is: A=a1 k1+a2 (1-k1) Where a1 is the asymmetric contrastive loss function, a2 is the cross-modal ordered pairing loss function, and k1 is the weight corresponding to the asymmetric contrastive loss function; The asymmetric contrastive loss function is: in, For temperature parameters, , Image features representing the first image, represent The transpose of the matrix, The image features representing the j-th first image, represent The transpose of the matrix, Represents the actual label value. B represents the text features with true label values, and B is the number of the first images input into the ordered learning model in each batch. The cross-modal ordered pairing loss function is: in, These are hyperparameters for the computation process. For the text feature described by the j-th text, For image features with true label values, This represents a label-based distance-aware function.

5. The image ordered classification model training method as described in claim 1, characterized in that, The loss function corresponding to the image ordered classification model is: B=b1 k2+b2 (1-k2) Where b1 is the cross-entropy loss function, b2 is the cross-modal ordered pairing loss function, and k2 is the weight corresponding to the cross-entropy loss function.

6. An image ordered classification model training apparatus, used to implement the image ordered classification model training method according to any one of claims 1 to 5, characterized in that, include: The data acquisition module is used to acquire multiple types of training samples, each training sample containing a text description and a first image corresponding to the text description; The data processing module is used to perform word segmentation on the text description to obtain word segments for each text description; to perform a first sorting process on the first word segments in each text description, and to sort each text description based on the result of the first sorting process; wherein, the first word segment is the word segment used to identify the training sample type in each text description; and to perform a second sorting process on the second word segments in each text description; wherein, the second word segment is the word segment in each text description other than the first word segment. The model training module is used to train an image ordered classification model based on the text descriptions after the first and second sorting processes and the first image.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images