Long-tail target data classification method and device for medical data

Abstract

The present invention provides a long-tail target data classification method and device for medical data, the method comprising: obtaining a target image to be classified in a long-tail target database, adding a category text description to the target image to obtain input data; inputting the input data into a pre-trained classification recognition model to obtain a classification result output by the classification recognition model; wherein the classification recognition model is obtained by training using sample data and a pre-built network model; the sample data is obtained by adding a category text description to a collected natural image, and the network model comprises a pre-built encoder, an image feature classification head, and a text-image guided image feature classification head. The method solves the problem of poor accuracy and generalization of long-tail classification in the prior art, and improves the accuracy and generalization of long-tail classification.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data processing, and in particular to a method and device for classifying long-tail target data of medical data. Background Art

[0002] In the process of medical data processing, such as in the case of malarial parasite classification or in medical image classification tasks such as blood cell classification, there is a common situation where the total amount of data is small (small sample) and the amount of data is unbalanced. Data imbalance refers to the fact that in the classification process, the proportion of the amount of data in each category of the data set used for network training is seriously unbalanced, where the category with a large amount of data is called the head category, the category with a moderate amount of data is called the middle category, and the category with a small amount of data is called the tail category.

[0003] At the same time, with the rapid development of deep learning networks, long-tail target classification methods often adopt strategies such as resampling, reweighting, transfer learning or multi-expert models to improve the generalization ability of the model for long-tail distribution data sets. However, existing methods are limited to relying solely on image modal information to solve long-tail classification problems, resulting in poor generalization ability of the model and low classification accuracy.

[0004] In view of this, a method and device for long-tail target data classification of medical data are provided to solve the problem of poor accuracy and generalization of long-tail classification in the prior art. Summary of the invention

[0005] The present invention provides a long-tail target data classification method and device for medical data, so as to solve the problem of poor accuracy and generalization of long-tail classification in the prior art and improve the accuracy and generalization of long-tail classification.

[0006] The present invention provides a long-tail target data classification method for medical data, the method comprising:

[0007] Obtaining a target image to be classified in a long-tail target database, and adding a category text description to the target image to obtain input data;

[0008] Inputting the input data into a pre-trained classification recognition model to obtain a classification result output by the classification recognition model;

[0009] Among them, the classification recognition model is trained using sample data and a pre-built network model; the sample data is obtained by adding category text descriptions to the collected natural images, and the network model includes a pre-built encoder, an image feature classification head, and a text-image guided image feature classification head.

[0010] In some embodiments, the training process of the classification recognition model includes:

[0011] Establish a dataset with long-tail distribution in natural scenes, add text descriptions to each category in the dataset, and divide the dataset into a training set, a validation set, and a test set;

[0012] Build the encoder;

[0013] An image feature classification head and a text-guided image feature classification head are established respectively, and a training set, a validation set and a test set are used to train, validate and test the network model respectively to obtain the classification recognition model.

[0014] In some embodiments, establishing an encoder specifically includes:

[0015] Build an image encoder;

[0016] Adopt the BPE algorithm and construct the vocabulary based on the training corpus;

[0017] Encoding and decoding the vocabulary to establish a corresponding relationship between the vocabulary and the word embedding vector;

[0018] Performing word segmentation and vectorization on the input text sentence, performing matching calculation on each word in the text sentence according to the vocabulary to complete the word segmentation of the input text sentence, and performing vectorization processing on the text sentence by using the correspondence between the vocabulary and the word embedding vector;

[0019] Based on the vectorized text sentences, a text encoder is established.

[0020] In some embodiments, the BPE algorithm is used to construct a vocabulary based on the training corpus, specifically including:

[0021] Get training corpus;

[0022] Determine the desired vocabulary size;

[0023] Determine frequency threshold;

[0024] Split all words in the training corpus into character sequences and use these character sequences to build an initial vocabulary;

[0025] The frequency of consecutive byte pairs in the training corpus is counted, the byte pairs with the highest frequency are selected to merge into new subwords, and the vocabulary is updated until the vocabulary size reaches the set expectation or the iteration ends when the highest frequency of byte pairs is lower than the frequency threshold.

[0026] In some embodiments, the network structure of the image encoder specifically includes:

[0027] A Modified Res Net submodule, wherein the Modified Res Net submodule downsamples the input feature map by a factor of 2, and obtains a feature map downsampled by a factor of 32 for the input image as input information for the attention pooling layer by continuously stacking three submodules and combining convolution and pooling modules;

[0028] The attention pooling layer obtains global information by averaging the input information, and uses Transformer to realize the interaction between the global information and the input information of the pooling layer to obtain the input information of interest to the global information.

[0029] In some embodiments, an image feature classification head and a text-guided image feature classification head are established respectively, and a network model is trained, verified and tested respectively using a training set, a validation set and a test set to obtain the classification recognition model, specifically including:

[0030] Establishing and training an image feature classification head, and characterizing classification probabilities based on image features by training the image feature classification head;

[0031] Establish and train a text-guided image feature classification head. By training the text-guided image feature classification head, the classification probability is represented based on image and text features.

[0032] In the pre-training stage, the image encoder and text encoder are trained using the training set;

[0033] In the fine-tuning stage, the text encoder is frozen, the image encoder is fine-tuned through the training set, the image feature classification head and the text-guided image feature classification head are trained, and the model performance is verified and tested through the validation set and the test set respectively to obtain the classification recognition model. The present invention also provides a long-tail target data classification device for medical data, the device comprising:

[0034] A data acquisition unit is used to acquire a target image to be classified in a long-tail target database, and to add a category text description to the target image to obtain input data;

[0035] A result generating unit, used for inputting the input data into a pre-trained classification recognition model to obtain a classification result output by the classification recognition model;

[0036] Among them, the classification recognition model is trained using sample data and a pre-built network model; the sample data is obtained by adding category text descriptions to the collected natural images, and the network model includes a pre-built encoder, an image feature classification head, and a text-image guided image feature classification head.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described above when executed by a processor.

[0039] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method described above is implemented.

[0040] The long-tail target data classification method for medical data provided by the present invention obtains a target image to be classified in a long-tail target database, and adds a category text description to the target image to obtain input data; the input data is input into a pre-trained classification recognition model to obtain a classification result output by the classification recognition model; wherein the classification recognition model is trained using sample data and a pre-constructed network model; the sample data is obtained by adding a category text description to a collected natural image, and the network model includes a pre-constructed encoder, an image feature classification head, and a text-image guided image feature classification head.

[0041] In this way, the method and device effectively utilize the relationship between image features and text features learned by the pre-training model, and then combine image features and text features in the fine-tuning stage to improve the accuracy performance on the long-tail classification task; in addition, in the inference stage, only the image encoder, image feature classification head and text-guided image feature classification head are retained, the network structure is streamlined, and the speed performance on the long-tail classification task is maintained. The problem of poor accuracy and generalization of long-tail classification in the prior art is solved, and the accuracy and generalization of long-tail classification are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is one of the flow charts of the long-tail target data classification method for medical data provided by the present invention;

[0044] Figure 2 This is the second flowchart of the long-tail target data classification method for medical data provided by the present invention;

[0045] Figure 3 This is the third flow chart of the long-tail target data classification method for medical data provided by the present invention;

[0046] Figure 4 It is a structural block diagram of a long-tail target data classification device for medical data provided by the present invention;

[0047] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The long-tail target data classification method for medical data provided by the present invention realizes accurate classification of long-tail distribution data in natural scenes by establishing a connection between image modality and text modality.

[0050] In a specific embodiment, the present invention provides a long-tail target data classification method for medical data, such as Figure 1 As shown, the method comprises the following steps:

[0051] S110: Obtain a target image to be classified in a long-tail target database, and add a category text description to the target image to obtain input data;

[0052] S120: Inputting the input data into a pre-trained classification recognition model to obtain a classification result output by the classification recognition model;

[0053] The classification recognition model is obtained by training with sample data and a pre-built network model; the sample data is obtained by adding category text descriptions to the collected natural images, such as Figure 3 As shown, the network model includes a pre-built encoder, an image feature classification head, and a text-image guided image feature classification head.

[0054] In some embodiments, Figure 2 As shown, the training process of the classification recognition model includes the following steps:

[0055] S210: Establishing a data set with a long-tail distribution in a natural scene, adding a text description for each category in the data set, and dividing the data set into a training set, a validation set, and a test set;

[0056] S220: Establish an encoder;

[0057] S230: Establishing an image feature classification head and a text-guided image feature classification head respectively, and using a training set, a validation set and a test set to respectively train, validate and test the network model to obtain the classification recognition model.

[0058] After the classification and recognition model is trained, the image data to be tested, i.e. the test set, is input into the classification and recognition model, and the classification result can be output to complete the classification task.

[0059] In step S220, an encoder is established, which specifically includes the following steps:

[0060] S310: Establish an image encoder; establish an image encoder, the image encoder combines Modified Res Net and an attention pooling layer, the Modified Res Net submodule can downsample the input feature map by 2 times, and by continuously stacking 3 submodules and combining convolution and pooling modules, a feature map of the input image downsampled by 32 times is obtained as input information of the attention pooling layer; the attention pooling layer obtains global information by averaging the input information, and uses Transformer to realize the interaction between the global information and the input information of the pooling layer, thereby obtaining input information of interest to the global information;

[0061] S320: Adopt the BPE algorithm to construct a vocabulary based on the training corpus; specifically, obtain the training corpus, determine the expected vocabulary size, and determine the frequency threshold; split all the words in the training corpus into character sequences, and use these character sequences to construct an initial vocabulary; count the frequency of occurrence of consecutive byte pairs in the training corpus, select the byte pairs with the highest occurrence frequency to merge into new sub-words, and update the vocabulary until the vocabulary size reaches the set expectation or the iteration ends when the highest occurrence frequency of the byte pairs is lower than the frequency threshold, and the frequency threshold can be 1.

[0062] S330: Encode and decode the vocabulary to establish a corresponding relationship between the vocabulary and the word embedding vector; when encoding and decoding the vocabulary, based on the vocabulary established in step S320, additionally introduce , and Marking bit, improves the generalization ability of the model for downstream tasks, and establishes the correspondence between the vocabulary and the word embedding vector based on the vocabulary.

[0063] S340: Segment and vectorize the input text sentence, perform matching calculation on each word in the text sentence according to the vocabulary to complete the segmentation of the input text sentence, and use the correspondence between the vocabulary and the word embedding vector to vectorize the text sentence. Segment and vectorize the input text sentence, and traverse each word in the input text sentence according to the vocabulary established in step S320 and find the vocabulary unit that matches it. If the vocabulary unit fails to match, match it. Marking position; introduce the hyperparameter context_length to control the length of the sequence after the text sentence is segmented, and introduce it at the beginning and end of the sequence respectively and Mark the position and complete the word segmentation; according to the correspondence between the word list and the word embedding vector established in step S330, the text sentence is vectorized.

[0064] S350: Based on the vectorized text sentences, a text encoder is established. Specifically, the text encoder uses Transformer to realize the information interaction between word embedding vectors. Its input combines word embedding vectors and position information. The masked self-attention mechanism realizes the interaction between text information and its left text information. Mark bit output text features.

[0065] In some embodiments, an image feature classification head and a text-guided image feature classification head are established respectively, and a network model is trained, verified and tested respectively using a training set, a validation set and a test set to obtain the classification recognition model, specifically including:

[0066] Establishing and training an image feature classification head, and characterizing classification probabilities based on image features by training the image feature classification head;

[0067] Establish and train a text-guided image feature classification head. By training the text-guided image feature classification head, the classification probability is represented based on image and text features.

[0068] In the pre-training stage, the image encoder and text encoder are trained using the training set;

[0069] In the fine-tuning stage, the text encoder is frozen, the image encoder is fine-tuned using the training set, the image feature classification head and the text-guided image feature classification head are trained, and the model performance is verified and tested using the validation set and the test set, respectively, to obtain the classification recognition model.

[0070] For ease of understanding, the following takes a specific usage scenario as an example to briefly describe the process of establishing a network model. The process of establishing a network model includes the following steps:

[0071] 1) Establish image encoder and text encoder separately and pre-train them, aiming to establish the relationship between image features and text features at the category level through pre-training. Specifically, first randomly sample a batch of images , matching the corresponding text description , represents the batch size; then, the image and text description are sent to the image encoder and text encoder , we get the image features and text features, which can be expressed as:

[0072] ;

[0073] In formulas (1) and (2), and Respectively represent images and text description The corresponding image features and text features, and and for dimension;

[0074] Based on the category level, contrastive learning is used to establish the relationship between image information and text description, which can be expressed as:

[0075] ;

[0076] In formula (3)-(5) and Represent the loss of visual and textual perspectives, respectively. represents the total loss, express and The cosine similarity of and Respectively and A subset of batch size and A text description and image of the corresponding category;

[0077] Finally, using the training set, validation set, and test set, combined The loss completes the pre-training of the model and obtains the pre-trained model;

[0078] In the fine-tuning stage, the text encoder is frozen, the image encoder is fine-tuned, and the image feature classification head and the text-guided image feature classification head are established and trained.

[0079] Specifically, the image feature classification head and the text-guided image feature classification head are designed to represent the probability distribution based on image information and text information. With text features interaction, thereby obtaining The following are the text information of the category you are interested in: represents the number of categories, Represents the maximum number of text sentences for all categories, Represent the dimensions of image features and text features, and then perform the classification task by integrating image information and text information, which can be expressed as:

[0080] ;

[0081] In formula (6)-(12), , , Represents the query, key, and value of the attention operation respectively; express Text information of the category of interest; represents a fully connected layer; express function; Represents the predicted probability distribution based on image information; express and The cosine similarity of is a constant; Represents the predicted probability distribution combining image information and text information;

[0082] In the fine-tuning stage, the loss function can be expressed as and The cross entropy loss can be expressed as:

[0083] ;

[0084] In formula (13), Indicates label information; represents the cross entropy loss function;

[0085] Using the training set and the validation set, combined The loss completes the fine-tuning of the model and obtains the classification model.

[0086] In the above specific implementation manner, the long-tail target data classification method for medical data provided by the present invention obtains the target image to be classified in the long-tail target database, and adds a category text description to the target image to obtain input data; the input data is input into a pre-trained classification recognition model to obtain the classification result output by the classification recognition model; wherein the classification recognition model is trained using sample data and a pre-constructed network model; the sample data is obtained by adding a category text description to the collected natural image, and the network model includes a pre-constructed encoder, an image feature classification head, and a text-image guided image feature classification head.

[0087] In this way, this method effectively utilizes the relationship between image features and text features learned by the pre-training model, and then combines image features and text features in the fine-tuning stage to improve the accuracy performance on the long-tail classification task; in addition, in the inference stage, only the image encoder, text encoder and text-guided image feature classification head are retained, which simplifies the network structure and maintains the speed performance on the long-tail classification task. This solves the problem of poor accuracy and generalization of long-tail classification in the existing technology and improves the accuracy and generalization of long-tail classification.

[0088] In addition to the above method, the present invention also provides a long-tail target data classification device for medical data, such as Figure 4 As shown, the device comprises:

[0089] The data acquisition unit 410 is used to acquire a target image to be classified in the long-tail target database, and add a category text description to the target image to obtain input data;

[0090] A result generating unit 420 is used to input the input data into a pre-trained classification recognition model to obtain a classification result output by the classification recognition model;

[0091] Among them, the classification recognition model is trained using sample data and a pre-built network model; the sample data is obtained by adding category text descriptions to the collected natural images, and the network model includes a pre-built encoder, an image feature classification head, and a text-image guided image feature classification head.

[0092] In some embodiments, the training process of the classification recognition model includes:

[0093] Establish a dataset with long-tail distribution in natural scenes, add text descriptions to each category in the dataset, and divide the dataset into a training set, a validation set, and a test set;

[0094] Build the encoder;

[0095] An image feature classification head and a text-guided image feature classification head are established respectively, and a training set, a validation set and a test set are used to train, validate and test the network model respectively to obtain the classification recognition model.

[0096] In some embodiments, establishing an encoder specifically includes:

[0097] Build an image encoder;

[0098] Adopt the BPE algorithm and construct the vocabulary based on the training corpus;

[0099] Encoding and decoding the vocabulary to establish a corresponding relationship between the vocabulary and the word embedding vector;

[0100] Performing word segmentation and vectorization on the input text sentence, performing matching calculation on each word in the text sentence according to the vocabulary to complete the word segmentation of the input text sentence, and performing vectorization processing on the text sentence by using the correspondence between the vocabulary and the word embedding vector;

[0101] Based on the vectorized text sentences, a text encoder is established.

[0102] In some embodiments, the BPE algorithm is used to construct a vocabulary based on the training corpus, specifically including:

[0103] Get training corpus;

[0104] Determine the desired vocabulary size;

[0105] Determine frequency threshold;

[0106] Split all words in the training corpus into character sequences and use these character sequences to build an initial vocabulary;

[0107] The frequency of consecutive byte pairs in the training corpus is counted, the byte pairs with the highest frequency are selected to merge into new subwords, and the vocabulary is updated until the vocabulary size reaches the set expectation or the iteration ends when the highest frequency of byte pairs is lower than the frequency threshold.

[0108] In some embodiments, the network structure of the image encoder specifically includes:

[0109] A Modified Res Net submodule, wherein the Modified Res Net submodule downsamples the input feature map by a factor of 2, and obtains a feature map downsampled by a factor of 32 for the input image as input information for the attention pooling layer by continuously stacking three submodules and combining convolution and pooling modules;

[0110] The attention pooling layer obtains global information by averaging the input information, and uses Transformer to realize the interaction between the global information and the input information of the pooling layer to obtain the input information of interest to the global information.

[0111] In some embodiments, an image feature classification head and a text-guided image feature classification head are established respectively, and a network model is trained, verified and tested respectively using a training set, a validation set and a test set to obtain the classification recognition model, specifically including:

[0112] Establishing and training an image feature classification head, and characterizing classification probabilities based on image features by training the image feature classification head;

[0113] Establish and train a text-guided image feature classification head. By training the text-guided image feature classification head, the classification probability is represented based on image and text features.

[0114] In the pre-training stage, the image encoder and text encoder are trained using the training set;

[0115] In the fine-tuning stage, the text encoder is frozen, the image encoder is fine-tuned using the training set, the image feature classification head and the text-guided image feature classification head are trained, and the model performance is verified and tested using the validation set and the test set, respectively, to obtain the classification recognition model.

[0116] In the above-mentioned specific embodiment, the long-tail target data classification device for medical data provided by the present invention obtains the target image to be classified in the long-tail target database, and adds a category text description to the target image to obtain input data; the input data is input into a pre-trained classification recognition model to obtain the classification result output by the classification recognition model; wherein the classification recognition model is trained using sample data and a pre-constructed network model; the sample data is obtained by adding a category text description to the collected natural image, and the network model includes a pre-constructed encoder, an image feature classification head, and a text-image guided image feature classification head.

[0117] In this way, the device effectively utilizes the relationship between image features and text features learned by the pre-training model, and then combines image features and text features in the fine-tuning stage to improve the accuracy performance on the long-tail classification task; in addition, in the inference stage, only the image encoder, image feature classification head and text-guided image feature classification head are retained, which simplifies the network structure and maintains the speed performance on the long-tail classification task. The problem of poor accuracy and generalization of long-tail classification in the prior art is solved, and the accuracy and generalization of long-tail classification are improved.

[0118] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the above method.

[0119] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0120] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the above method.

[0121] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A long-tail target data classification method for medical data, It is characterized in that The method comprises: Obtaining a target image to be classified in a long-tail target database, and adding a category text description to the target image to obtain input data; Inputting the input data into a pre-trained classification recognition model to obtain a classification result output by the classification recognition model; The classification recognition model is obtained by training using sample data and a pre-built network model; the sample data is obtained by adding category text descriptions to collected natural images, and the network model includes a pre-built encoder, an image feature classification head, and a text-image guided image feature classification head; The training process of the classification recognition model includes: Establish a dataset with long-tail distribution in natural scenes, add text descriptions to each category in the dataset, and divide the dataset into a training set, a validation set, and a test set; Build the encoder; An image feature classification head and a text-guided image feature classification head are established respectively, and the network model is trained, verified and tested respectively using a training set, a validation set and a test set to obtain the classification recognition model, specifically including: Establishing and training an image feature classification head, and characterizing classification probabilities based on image features by training the image feature classification head; Establish and train a text-guided image feature classification head. By training the text-guided image feature classification head, the classification probability is represented based on image and text features. In the pre-training stage, the image encoder and text encoder are trained through the training set; the relationship between image features and text features is established at the category level through pre-training. Pre-training specifically includes: randomly sampling a batch of images , matching the corresponding text description , represents the batch size; image and text description are sent to the image encoder and text encoder , we get the image features and text features, expressed as: ； In formulas (1) and (2), and Respectively represent images and text description The corresponding image features and text features; Based on the category level, contrastive learning is used to establish the relationship between image information and text description, which can be expressed as: ； In formula (3)-(5) and Represent the loss of image and text angles respectively, represents the total loss, express and The cosine similarity of and Respectively and A subset of , which is the batch size and The image and text description of the corresponding category; In the fine-tuning stage, the text encoder is frozen, the image encoder is fine-tuned using the training set, the image feature classification head and the text-guided image feature classification head are trained, and the model performance is verified and tested using the validation set and the test set, respectively, to obtain the classification recognition model.

2. The long-tail target data classification method for medical data according to claim 1, It is characterized in that Build the encoder, including: Build an image encoder; Adopt the BPE algorithm and construct the vocabulary based on the training corpus; Encoding and decoding the vocabulary to establish a corresponding relationship between the vocabulary and the word embedding vector; Performing word segmentation and vectorization on the input text sentence, performing matching calculation on each word in the text sentence according to the vocabulary to complete the word segmentation of the input text sentence, and performing vectorization processing on the text sentence by using the correspondence between the vocabulary and the word embedding vector; Based on the vectorized text sentences, a text encoder is established.

3. The long-tail target data classification method for medical data according to claim 2, It is characterized in that Using the BPE algorithm, a vocabulary is constructed based on the training corpus, including: Get training corpus; Determine the desired vocabulary size; Determine frequency threshold; Split all words in the training corpus into character sequences and use these character sequences to build an initial vocabulary; The frequency of consecutive byte pairs in the training corpus is counted, the byte pairs with the highest frequency are selected to merge into new subwords, and the vocabulary is updated until the vocabulary size reaches the set expectation or the iteration ends when the highest frequency of byte pairs is lower than the frequency threshold.

4. The long-tail target data classification method for medical data according to claim 2, It is characterized in that The network structure of the image encoder specifically includes: A Modified Res Net submodule, wherein the Modified Res Net submodule downsamples the input feature map by a factor of 2, and obtains a feature map downsampled by a factor of 32 for the input image as input information for the attention pooling layer by continuously stacking three submodules and combining convolution and pooling modules; The attention pooling layer obtains global information by averaging the input information, and uses Transformer to realize the interaction between the global information and the input information of the pooling layer to obtain the input information of interest to the global information.

5. A long-tail target data classification device for medical data, used to implement the method according to any one of claims 1 to 4, It is characterized in that The device comprises: A data acquisition unit is used to acquire a target image to be classified in a long-tail target database, and to add a category text description to the target image to obtain input data; A result generating unit, used for inputting the input data into a pre-trained classification recognition model to obtain a classification result output by the classification recognition model; Among them, the classification recognition model is trained using sample data and a pre-built network model; the sample data is obtained by adding category text descriptions to the collected natural images, and the network model includes a pre-built encoder, an image feature classification head, and a text-image guided image feature classification head.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images