Image category prediction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image category prediction method and device, electronic equipment and a storage medium, belongs to the technical field of image processing, and is suitable for the field of medical science and technology. The method comprises the steps that a sample image marked with a reference image category is acquired, image-text conversion is carried out on the sample image, an image description text is obtained, and the reference image category is used for representing the category of the sample image; obtaining a target image, and performing image-text conversion on the target image to obtain a target description text; performing similarity calculation on the image description text and the target description text to obtain a text similarity; performing feature extraction on the target image to obtain image features; performing similarity calculation on the image features and the reference image category to obtain category similarity; and based on the text similarity and the category similarity, performing category prediction on the target image. According to the embodiment of the invention, the accuracy of image category prediction can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, is applicable to the field of medical technology, and in particular to an image category prediction method and device, an electronic device, and a storage medium. Background Art

[0002] Image category prediction refers to predicting the image category of image data. For example, in the lung disease prediction scenario in the medical field, the lung status category of the patient's lung image is predicted to obtain the lung status category in the lung image. The doctor can determine whether the patient has pneumonia based on the status category, thereby helping the doctor to quickly understand the patient's lung condition and improve the doctor's diagnostic efficiency.

[0003] At present, common image category prediction methods are mostly to extract image features of image data and then classify the image data according to the image features. This method is easily affected by image noise, resulting in image classification prediction errors. Therefore, how to improve the accuracy of image category prediction has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a face mask detection method and device, an electronic device and a storage medium, aiming to improve the accuracy of image category prediction.

[0005] To achieve the above objective, a first aspect of an embodiment of the present application proposes an image category prediction method, the method comprising:

[0006] Acquire a sample image annotated with a reference image category, and perform image-to-text conversion on the sample image to obtain an image description text, wherein the reference image category is used to indicate the category of the sample image;

[0007] Acquire a target image, and perform image-to-text conversion on the target image to obtain a target description text;

[0008] Calculating the similarity between the image description text and the target description text to obtain text similarity;

[0009] Extracting features from the target image to obtain image features;

[0010] Calculating similarity between the image features and the reference image category to obtain category similarity;

[0011] Based on the text similarity and the category similarity, a category prediction is performed on the target image.

[0012] In some embodiments, the calculating the similarity between the image description text and the target description text to obtain the text similarity includes:

[0013] Performing word segmentation processing on the image description text to obtain image description phrases, and performing word segmentation processing on the target description text to obtain target description phrases;

[0014] removing stop words from the image description phrase to obtain an image description vocabulary, and removing stop words from the target description phrase to obtain a target description vocabulary;

[0015] Calculating the vocabulary similarity of the image description vocabulary and the target description vocabulary to obtain initial text similarity;

[0016] The initial text similarity is normalized to obtain the text similarity.

[0017] In some embodiments, the calculating of vocabulary similarity between the image description vocabulary and the target description vocabulary to obtain initial text similarity includes:

[0018] De-duplicating the image description vocabulary to obtain an image de-duplicated vocabulary, and de-duplicating the target description vocabulary to obtain a target de-duplicated vocabulary;

[0019] Performing repeated word collection on the image deduplication words and the target deduplication words to obtain description intersection words;

[0020] Merging the image description vocabulary and the target description vocabulary to obtain a description merged vocabulary;

[0021] De-duplication of the description combined vocabulary is performed to obtain a description union vocabulary;

[0022] Calculating the word length of the description intersection words to obtain a first word count, and calculating the word length of the description union words to obtain a second word count;

[0023] A quotient operation is performed on the first vocabulary number and the second vocabulary number to obtain the initial text similarity.

[0024] In some embodiments, the step of obtaining a sample image labeled with a reference image category and converting the sample image into text to obtain an image description text includes:

[0025] Acquire an initial image, and perform image category labeling on the initial image to obtain the sample image;

[0026] Dividing the sample image into blocks to obtain image blocks;

[0027] Extracting features from the image block to obtain block features;

[0028] Based on the image block features, a text description is performed on the image block to obtain an image block description text;

[0029] The image description text is merged to obtain the image description text.

[0030] In some embodiments, the performing text description on the image block based on the image block feature to obtain the image block description text includes:

[0031] Performing feature projection on the block features to obtain text features;

[0032] The text feature is subjected to text decoding to obtain the image block description text.

[0033] In some embodiments, the performing similarity calculation on the image feature and the reference image category to obtain category similarity includes:

[0034] Encoding the reference image category to obtain a category identification vector;

[0035] Performing cosine similarity calculation on the image feature and the category identification vector to obtain the initial category similarity;

[0036] The initial category similarity is normalized to obtain the category similarity.

[0037] In some embodiments, the performing category prediction on the target image based on the text similarity and the category similarity includes:

[0038] Performing similarity evaluation on the text similarity to obtain text similarity evaluation data;

[0039] Performing similarity evaluation on the category similarity to obtain category similarity evaluation data;

[0040] Merging the text similarity evaluation data and the category similarity evaluation data to obtain a target similarity;

[0041] Based on the target similarity, the reference image category is screened to obtain an image prediction category.

[0042] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image category prediction device, the device comprising:

[0043] A sample text description module is used to obtain a sample image marked with a reference image category, perform image-text conversion on the sample image, and obtain an image description text, wherein the reference image category is used to indicate the category of the sample image;

[0044] A target text description module is used to obtain a target image, perform image-text conversion on the target image, and obtain a target description text;

[0045] A text similarity calculation module is used to calculate the similarity between the image description text and the target description text to obtain text similarity;

[0046] An image feature extraction module is used to extract features from the target image to obtain image features;

[0047] A category similarity calculation module, used to perform similarity calculation on the image features and the reference image category to obtain category similarity;

[0048] The image category prediction module is used to predict the category of the target image based on the text similarity and the category similarity.

[0049] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0050] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0051] The image category prediction method and device, electronic device and storage medium proposed in the present application obtain image description text by performing image-text conversion on a sample image, and obtain target description text by performing image-text conversion on a target image, so as to facilitate the extraction of semantic information from the text and reduce the influence of image noise on image type prediction. Furthermore, by performing similarity calculation on the image description text and the target description text, it is convenient to evaluate the semantic similarity between the target image and the sample image, thereby improving the accuracy of image type prediction. Secondly, by performing feature extraction on the target image to obtain image features, and then performing similarity calculation on the image features and the reference image categories to obtain category similarity, it is convenient to evaluate the degree of matching between the target image and the reference image categories from the image feature level. Finally, according to the text similarity and the category similarity, the category of the target image is predicted to achieve a comprehensive prediction of the target image, making the image category prediction more comprehensive, thereby improving the accuracy of image category prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flowchart of an image category prediction method provided in an embodiment of the present application;

[0053] Figure 2yes Figure 1 Flow chart of step S101 in FIG.

[0054] Figure 3 yes Figure 2 Flow chart of step S204 in FIG.

[0055] Figure 4 yes Figure 1 Flow chart of step S103 in FIG.

[0056] Figure 5 yes Figure 4 Flow chart of step S403 in FIG.

[0057] Figure 6 yes Figure 1 Flow chart of step S105 in FIG.

[0058] Figure 7 yes Figure 1 Flow chart of step S106 in FIG.

[0059] Figure 8 is a schematic diagram of the structure of an image category prediction device provided in an embodiment of the present application;

[0060] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] First, some nouns involved in this application are analyzed:

[0065] Jaccard similarity coefficient: Jaccard similarity coefficient is a statistical method to measure the similarity between two sets. The similarity is determined by calculating the ratio of the size of the intersection of the two sets to the size of the union. In text analysis, Jaccard similarity is often used to compare the similarity of documents or text fragments by comparing the intersection and union of their word frequency vectors. The value of Jaccard similarity is between 0 and 1. The closer the value is to 1, the more similar the two sets are.

[0066] Cosine similarity: Cosine similarity is a metric commonly used to measure the similarity of two vectors in direction rather than their magnitude. It is calculated by calculating the ratio of the dot product of two vectors to the product of their moduli. The value range is between -1 and 1, where 1 means the vectors are exactly the same in direction, -1 means they are completely opposite, and 0 means there is no correlation between the vectors. This similarity measure is widely used in text analysis, information retrieval, and recommendation systems, especially when dealing with high-dimensional data, because it can ignore the magnitude changes of vectors and only focus on the relative direction between vectors.

[0067] Image category prediction refers to predicting the image category of image data. For example, in the lung disease prediction scenario in the medical field, the lung status category of the patient's lung image is predicted to obtain the lung status category in the lung image. The doctor can determine whether the patient has pneumonia based on the status category, thereby helping the doctor to quickly understand the patient's lung condition and improve the doctor's diagnostic efficiency.

[0068] At present, common image category prediction methods are mostly to extract image features of image data and then classify the image data according to the image features. This method is easily affected by image noise, resulting in image classification prediction errors. Therefore, how to improve the accuracy of image category prediction has become a technical problem that needs to be solved urgently.

[0069] Based on this, the embodiments of the present application provide an image category prediction method and device, an electronic device and a storage medium, aiming to improve the accuracy of image category prediction.

[0070] The image category prediction method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the image category prediction method in the embodiments of the present application is described.

[0071] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0072] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0073] The image category prediction method provided in the embodiment of the present application relates to the field of image processing technology and is applicable to the field of medical technology. The image category prediction method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image category prediction method, etc., but is not limited to the above forms.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] Figure 1 is an optional flowchart of the image category prediction method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0076] Step S101, obtaining a sample image annotated with a reference image category, performing image-to-text conversion on the sample image to obtain an image description text, wherein the reference image category is used to indicate the category of the sample image;

[0077] Step S102, obtaining a target image, performing image-to-text conversion on the target image, and obtaining a target description text;

[0078] Step S103, calculating the similarity between the image description text and the target description text to obtain text similarity;

[0079] Step S104, extracting features from the target image to obtain image features;

[0080] Step S105, calculating the similarity between the image features and the reference image categories to obtain category similarity;

[0081] Step S106: predicting the category of the target image based on the text similarity and the category similarity.

[0082] In steps S101 to S106 shown in the embodiment of the present application, image description text is obtained by performing image-text conversion on a sample image marked with a reference image category, wherein the reference image category is used to represent the category of the sample image. Furthermore, a target description text is obtained by performing image-text conversion on the acquired target image. Next, similarity is calculated between the image description text and the target description text to obtain text similarity. Feature extraction is then performed on the target image to obtain image features. Similarity is calculated between the image features and the reference image category to obtain category similarity. Finally, category prediction is performed on the target image based on the text similarity and category similarity. Therefore, the present application performs image-text conversion on the sample image to obtain image description text, and performs image-text conversion on the target image to obtain target description text, so as to facilitate the extraction of semantic information from the text and reduce the influence of image noise on image type prediction. Furthermore, by performing similarity calculation on the image description text and the target description text, it is convenient to evaluate the semantic similarity between the target image and the sample image, thereby improving the accuracy of image type prediction. Secondly, by performing feature extraction on the target image to obtain image features, and then performing similarity calculation on the image features and the reference image categories to obtain category similarity, it is convenient to evaluate the degree of matching between the target image and the reference image categories from the image feature level. Finally, according to the text similarity and category similarity, the target image is predicted to achieve a comprehensive prediction of the target image, making the image category prediction more comprehensive, thereby improving the accuracy of image category prediction.

[0083] In step S101 of some embodiments, the sample images refer to a set of images having all image categories within a specific field. For example, in the field of lung medicine, the sample images may be a set of images having image categories that may include pneumonia image categories, lung tumor image categories, normal lung image categories, and the like.

[0084] The present application can obtain initial images in a certain field by means of network search or database query, and then label the initial images by categories to obtain reference image categories. Secondly, the initial images labeled with reference image categories are aggregated to obtain sample images. Finally, image features in the sample images are described to obtain image description text.

[0085] For details, see Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S205:

[0086] Step S201, obtaining an initial image, and labeling the initial image by image category to obtain a sample image;

[0087] Step S202, dividing the sample image into blocks to obtain image blocks;

[0088] Step S203, extracting features from the image block to obtain block features;

[0089] Step S204, based on the image block features, a text description is performed on the image block to obtain the image block description text;

[0090] Step S205, merging the block description text to obtain the image description text.

[0091] In step S201 of some embodiments, the initial image refers to a set of images without labeled image categories. It should be noted that the initial image can be obtained through network search or database query. Furthermore, the initial image can be labeled with image categories through manual labeling to obtain sample images.

[0092] In step S202 of some embodiments, the sample image may be segmented into a plurality of image blocks of the same size by performing image segmentation on the sample image.

[0093] In step S203 of some embodiments, the image features of each image block, i.e., the block features, can be extracted by performing convolution pooling processing on each image block. The block features of the image block can also be extracted by traditional computer vision processing methods. It should be noted that the traditional computer vision processing method includes edge detection, corner detection, image texture analysis, shape analysis, color extraction, template matching and other steps.

[0094] In step S204 of some embodiments, the vector identifier of the tile feature may be mapped to the vector space of the text vector to obtain the text representation of the tile feature, that is, the tile description text.

[0095] For details, see Figure 3In some embodiments, step S204 may include but is not limited to steps S301 to S302:

[0096] Step S301, performing feature projection on the block features to obtain text features;

[0097] Step S302: decoding the text features to obtain the block description text.

[0098] In step S301 of some embodiments, the block features in the image feature vector space may be mapped to the text feature vector space through a preset mapping function, thereby obtaining a text vector representation of the block features, ie, the text features.

[0099] In step S302 of some embodiments, the text features are described in text form, thereby implementing text decoding of the text features and obtaining the image block description text.

[0100] In steps S301 to S302 shown in this embodiment, by performing feature projection on the block features to obtain text features, and then performing text decoding on the text features to obtain the block description text, the description accuracy of the image block can be improved, thereby improving the description accuracy of the sample image.

[0101] In step S205 of some embodiments, the image description text for describing the sample image is obtained by merging the image description text of each image block.

[0102] In steps S201 to S205 shown in the present embodiment, the acquired initial image is labeled with image categories to obtain a sample image, and then the sample image is segmented to obtain image blocks. Furthermore, feature extraction is performed on the image blocks to obtain block features. Next, based on the block features, a text description is performed on the image blocks to obtain block description text. Finally, the block description text is merged to obtain image description text. This can reduce the computational complexity of image feature extraction and improve the efficiency of image feature extraction. In addition, by performing feature extraction on each image block, the semantic loss of image-text conversion of the sample image can also be reduced, thereby improving the accuracy of the image description text.

[0103] In step S102 of some embodiments, the target image may be an image that has no labeled reference image category and needs to perform image category prediction.

[0104] The present application can obtain a target image by receiving an image uploaded by a user that needs to be predicted in terms of image category, and further obtain a target description text by performing image-to-text conversion processing on image features in the target image.

[0105] In detail, the method of converting the target image into text is similar to the method of converting the sample image into text, so it will not be described in detail.

[0106] In step S103 of some embodiments, the phrases in the image description text may be compared with the phrases in the target description text, thereby calculating the similarity between the image description text and the target description text.

[0107] For details, see Figure 4 In some embodiments, step S103 may include but is not limited to steps S401 to S404:

[0108] Step S401, performing word segmentation processing on the image description text to obtain image description phrases, and performing word segmentation processing on the target description text to obtain target description phrases;

[0109] Step S402, removing stop words from the image description phrase to obtain an image description vocabulary, and removing stop words from the target description phrase to obtain a target description vocabulary;

[0110] Step S403, calculating the vocabulary similarity of the image description vocabulary and the target description vocabulary to obtain the initial text similarity;

[0111] Step S404, normalizing the initial text similarity to obtain text similarity.

[0112] In step S401 of some embodiments, image description phrases can be obtained by decomposing the text strings in the above image description text into smaller units. At the same time, target description phrases can be obtained by decomposing the text strings in the above target description text into smaller units. For example, when the image description text is: "Chest X-ray, showing the outlines of the lungs and heart, the left lobe of the lung has obvious shadows, and pneumonia may exist"; the target description text is: "Chest X-ray, showing the outlines of the lungs and heart, the right lobe of the lung has shadows, and pneumonia may exist", the image description phrases can be: {"chest", "X-ray", "show", "show", "lungs", "and", "heart outline", "left lobe of the lung", "has", "obvious shadows", "may", "exists", "pneumonia"}, and the target description phrases can be: {"chest", "X-ray", "show", "show", "lungs", "and", "heart outline", "right lobe of the lung", "has", "shadows", "may", "exists", "pneumonia"}.

[0113] In step S402 of some embodiments, stop words refer to common words that are considered to not carry important value in text processing and information retrieval, such as punctuation marks, conjunctions, prepositions, pronouns, etc. For example, the above-mentioned image description phrases {"chest", "X-ray", "display", "had", "lungs", "and", "heart outline", "left lung lobe", "has", "obvious shadow", "may", "exist", "pneumonia"} and the target description phrases {"chest", "X-ray", "display", "had", "lungs", "and", "heart outline", "right lung lobe", "has", "shadow", "may", "exist", "pneumonia"} are removed from the stop words, and the image description vocabulary can be obtained as follows: {"chest", "X-ray", "lungs", "heart outline", "left lung lobe", "obvious shadow", "may", "exist", "pneumonia"} and the target description vocabulary is: {"chest", "X-ray", "lungs", "heart outline", "right lung lobe", "obvious shadow", "may", "exist", "pneumonia"}.

[0114] The present application can obtain image description vocabulary by removing phrases that are considered to carry no important value from image description phrases, and at the same time, can obtain target description vocabulary by removing phrases that are considered to carry no important value from target description phrases.

[0115] In step S403 of some embodiments, the similarity between the image description vocabulary and the target description vocabulary may be calculated according to the similarity idea of ​​the Jaccard similarity coefficient calculation method, thereby obtaining the initial text similarity between the image description text and the target description text.

[0116] For details, see Figure 5 In some embodiments, step S403 may include but is not limited to steps S501 to S506:

[0117] Step S501, deduplication of image description vocabulary to obtain image deduplication vocabulary, deduplication of target description vocabulary to obtain target deduplication vocabulary;

[0118] Step S502, performing duplicate word collection on the image deduplication words and the target deduplication words to obtain description intersection words;

[0119] Step S503, merging the image description vocabulary and the target description vocabulary to obtain a description merging vocabulary;

[0120] Step S504, deduplication of description merging words is performed to obtain description union words;

[0121] Step S505, calculating the word length of the description intersection words to obtain a first word count, and calculating the word length of the description union words to obtain a second word count;

[0122] Step S506, performing a quotient operation on the first vocabulary number and the second vocabulary number to obtain an initial text similarity.

[0123] In step S501 of some embodiments, by deduplicating repeated words in the image description vocabulary, an image deduplicated vocabulary can be obtained. At the same time, by deduplicating repeated words in the target description vocabulary, a target deduplicated vocabulary can be obtained. For example, the above-mentioned image description vocabulary {"chest", "X-ray", "lungs", "heart outline", "left lung lobe", "obvious shadow", "possible", "existence", "pneumonia"} and the target description vocabulary {"chest", "X-ray", "lungs", "heart outline", "right lung lobe", "obvious shadow", "possible", "existence", "pneumonia"} are deduplicated, and the image deduplicated vocabulary can be {"chest", "X-ray", "lungs", "heart outline", "left lung lobe", "obvious shadow", "possible", "existence", "pneumonia"}, and the target deduplicated vocabulary is {"chest", "X-ray", "lungs", "heart outline", "right lung lobe", "obvious shadow", "possible", "existence", "pneumonia"}.

[0124] In step S502 of some embodiments, the above-mentioned image deduplication vocabulary is merged with the repeated phrases in the above-mentioned target deduplication vocabulary to obtain a description merged vocabulary. For example, the above-mentioned image deduplication vocabulary {"chest", "X-ray", "lungs", "heart outline", "left lung lobe", "obvious shadow", "possible", "existence", "pneumonia"} and the repeated phrases in the target deduplication vocabulary {"chest", "X-ray", "lungs", "heart outline", "right lung lobe", "obvious shadow", "possible", "existence", "pneumonia"} are grouped to obtain the description merged vocabulary of {"chest", "X-ray", "lungs", "heart outline", "possible", "existence", "pneumonia"}.

[0125] In step S503 of some embodiments, the above-mentioned image description vocabulary is merged with the phrases in the above-mentioned target description vocabulary to obtain a description merged vocabulary. For example, the above-mentioned image description vocabulary {"chest", "X-ray", "lungs", "heart outline", "left lung lobe", "obvious shadow", "possible", "exist", "pneumonia"} and the target description vocabulary {"chest", "X-ray", "lungs", "heart outline", "right lung lobe", "obvious shadow", "possible", "exist", "pneumonia"} are merged to obtain a description merged vocabulary of {"chest", "chest", "X-ray", "X-ray", "lungs", "lungs", "heart outline", "heart outline", "left lung lobe", "obvious shadow", "right lung lobe", "shadow", "possible", "possible", "exist", "exist", "pneumonia", "pneumonia"}.

[0126] In step S504 of some embodiments, the description union vocabulary can be obtained by deduplicating repeated words in the description merge vocabulary. For example, the description merge vocabulary of {"chest", "chest", "X-ray", "X-ray", "lungs", "lungs", "heart outline", "heart outline", "left lobe of the lung", "obvious shadow", "right lobe of the lung", "shadow", "possible", "possible", "exists", "exists", "pneumonia", "pneumonia"} is deduplicated to obtain the description union vocabulary of {"chest", "X-ray", "lungs", "heart outline", "left lobe of the lung", "obvious shadow", "right lobe of the lung", "shadow", "possible", "exists", "pneumonia"}.

[0127] In step S505 of some embodiments, by calculating the number of words in the description intersection vocabulary, the vocabulary length of the description intersection vocabulary, that is, the first vocabulary number, can be obtained. By calculating the number of words in the description union vocabulary, the vocabulary length of the description union vocabulary, that is, the second vocabulary number, can be obtained. For example, the above description intersection vocabulary {"chest", "X-ray", "lungs", "heart outline", "possible", "existence", "pneumonia"} has 7 words, so the first vocabulary number is 7. The above description union vocabulary {"chest", "X-ray", "lungs", "heart outline", "left lobe of the lung", "obvious shadow", "right lobe of the lung", "shadow", "possible", "existence", "pneumonia"} has 11 words, so the second vocabulary number is 11.

[0128] In step S506 of some embodiments, the first vocabulary number is divided by the second vocabulary number to obtain the initial text similarity between the image description text and the target description text. For example, when the first vocabulary number is 7 and the second vocabulary number is 11, the initial text similarity is 7 / 11.

[0129] In steps S501 to S506 shown in the present embodiment, image description vocabulary is deduplicated to obtain image deduplicated vocabulary, target description vocabulary is deduplicated to obtain target deduplicated vocabulary, image deduplicated vocabulary and target deduplicated vocabulary are repeated to obtain description intersection vocabulary, further, image description vocabulary and target description vocabulary are merged to obtain description merged vocabulary, and description merged vocabulary is deduplicated to obtain description union vocabulary, then, description intersection vocabulary is calculated for vocabulary length to obtain a first vocabulary number, description union vocabulary is calculated for vocabulary length to obtain a second vocabulary number, and finally, the quotient operation is performed on the first vocabulary number and the second vocabulary number to obtain an initial text similarity, which can be used to evaluate the semantic similarity between the target image and the sample image, thereby achieving the purpose of improving the accuracy of image type prediction.

[0130] In step S404 of some embodiments, the text similarity is obtained by normalizing the initial text similarity, so that the user can clearly understand the similarity between the image description text and the target description text.

[0131] In steps S401 to S404 shown in the present embodiment, the image description text is segmented to obtain image description phrases, the target description text is segmented to obtain target description phrases, further, stop words are removed from the image description phrases to obtain image description vocabulary, stop words are removed from the target description phrases to obtain target description vocabulary, then vocabulary similarity is calculated for the image description vocabulary and the target description vocabulary to obtain initial text similarity, and finally, the initial text similarity is normalized to obtain text similarity, so as to facilitate the evaluation of the semantic similarity between the target image and the sample image, thereby improving the accuracy of image type prediction.

[0132] In step S104 of some embodiments, the target image may be subjected to convolution pooling processing to extract features of the target image, namely, image features. Alternatively, the image features of the target image may be extracted using traditional computer vision processing methods.

[0133] In step S105 of some embodiments, by calculating the similarity between the vector representing the reference image category and the vector representing the image feature, the category similarity between the image category of the target image and the reference image category can be obtained.

[0134] For details, see Figure 6 In some embodiments, step S105 may include but is not limited to steps S601 to S603:

[0135] Step S601, encoding the reference image category to obtain a category identification vector;

[0136] Step S602, performing cosine similarity calculation on the image feature and the category identification vector to obtain initial category similarity;

[0137] Step S603: normalize the initial category similarity to obtain the category similarity.

[0138] In step S601 of some embodiments, a category identification vector can be obtained by converting the reference image category into a vector representation.

[0139] In step S602 of some embodiments, multiple initial category similarities can be obtained by calculating the cosine similarity between the vector representation of the image feature and each category identification vector.

[0140] In step S603 of some embodiments, the category similarity between the image category of the target image and the category of the reference image can be obtained by normalizing the initial category similarity.

[0141] In steps S601 to S603 shown in the present embodiment, the reference image category is encoded to obtain a category identification vector, and then the cosine similarity between the image features and the category identification vector is calculated to obtain an initial category similarity. Furthermore, the initial category similarity is normalized to obtain a category similarity, which facilitates the evaluation of the degree of matching between the target image and the reference image category from the image feature level.

[0142] In step S106 of some embodiments, after obtaining the text similarity between the target image and the sample image, and the category similarity between the image features of the target image and the reference image category, the target similarity between the target image and the sample image can be calculated by weighted summation. Furthermore, based on the target similarity, the category of the target image can be predicted to obtain the image prediction category.

[0143] For details, see Figure 7 In some embodiments, step S106 may include but is not limited to steps S701 to S704:

[0144] Step S701, performing similarity evaluation on text similarity to obtain text similarity evaluation data;

[0145] Step S702, performing similarity evaluation on the category similarity to obtain category similarity evaluation data;

[0146] Step S703, merging the text similarity evaluation data and the category similarity evaluation data to obtain a target similarity;

[0147] Step S704: based on the target similarity, the reference image category is screened to obtain the image prediction category.

[0148] In step S701 and step S702 of some embodiments, the proportion score data of text similarity and category similarity are calculated respectively by pre-set proportion weights, so that text similarity evaluation data and category similarity evaluation data can be obtained. It should be noted that the sum of the proportion weights of text similarity and category similarity is 1.

[0149] In step S703 of some embodiments, the text similarity evaluation data and the category similarity evaluation data are combined by data addition, so as to obtain the image similarity between the target image and each sample image, that is, the target similarity.

[0150] In step S704 of some embodiments, based on the target similarity, a sample image that is most similar to the target image can be identified, and then the reference image category of the sample image is used as the image category of the target image to obtain the image prediction category.

[0151] In steps S701 to S704 shown in this embodiment, text similarity is evaluated to obtain text similarity evaluation data. Further, category similarity is evaluated to obtain category similarity evaluation data. Then, the text similarity evaluation data and the category similarity evaluation data are merged to obtain target similarity. Finally, according to the target similarity, the reference image category is screened to obtain the image prediction category. This can achieve comprehensive prediction of the target image, making the image category prediction more comprehensive, thereby improving the accuracy of the image category prediction.

[0152] The present application performs image-to-text conversion on a sample image to obtain an image description text, and performs image-to-text conversion on a target image to obtain a target description text, thereby facilitating the extraction of semantic information from the text and reducing the influence of image noise on image type prediction. Furthermore, by performing similarity calculation on the image description text and the target description text, it is convenient to evaluate the semantic similarity between the target image and the sample image, thereby improving the accuracy of image type prediction. Secondly, by performing feature extraction on the target image to obtain image features, and then performing similarity calculation on the image features and the reference image categories to obtain category similarity, it is convenient to evaluate the degree of matching between the target image and the reference image categories at the image feature level. Finally, based on the text similarity and the category similarity, the target image is predicted to achieve a comprehensive prediction of the target image, making the image category prediction more comprehensive, thereby improving the accuracy of image category prediction.

[0153] See also Figure 8The present application also provides an image category prediction device, which can implement the above-mentioned image category prediction method, and the device includes:

[0154] The sample text description module 801 is used to obtain a sample image marked with a reference image category, perform image-text conversion on the sample image, and obtain an image description text, wherein the reference image category is used to indicate the category of the sample image;

[0155] The target text description module 802 is used to obtain a target image, convert the target image into text, and obtain a target description text;

[0156] The text similarity calculation module 803 calculates the similarity of the image description text and the target description text to obtain the text similarity;

[0157] The image feature extraction module 804 is used to extract features from the target image to obtain image features;

[0158] The category similarity calculation module 805 is used to calculate the similarity between the image features and the reference image category to obtain the category similarity;

[0159] The image category prediction module 806 is used to predict the category of the target image based on the text similarity and the category similarity.

[0160] The specific implementation of the image category prediction device is substantially the same as the specific implementation of the above-mentioned image category prediction method, and will not be described in detail herein.

[0161] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image category prediction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0162] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0163] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0164] The memory 902 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the image category prediction method of the embodiment of this application;

[0165] Input / output interface 903, used to implement information input and output;

[0166] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0167] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0168] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0169] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image category prediction method is implemented.

[0170] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The image category prediction method, image category prediction device, electronic device and storage medium provided by the embodiments of the present application obtain image description text by performing image-text conversion on a sample image marked with a reference image category, wherein the reference image category is used to represent the category of the sample image. Furthermore, the target description text is obtained by performing image-text conversion on the acquired target image. Then, similarity is calculated between the image description text and the target description text to obtain text similarity. Feature extraction is then performed on the target image to obtain image features. Similarity is calculated between the image features and the reference image category to obtain category similarity. Finally, category prediction is performed on the target image based on the text similarity and category similarity.

[0172] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0173] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0174] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0176] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0177] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0178] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0182] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for predicting image categories, characterized in that: The method comprises: Acquire a sample image annotated with a reference image category, and perform image-to-text conversion on the sample image to obtain an image description text, wherein the reference image category is used to indicate the category of the sample image; Acquire a target image, and convert the target image into text to obtain a target description text; Calculating the similarity between the image description text and the target description text to obtain text similarity; Extracting features from the target image to obtain image features; Calculating similarity between the image features and the reference image category to obtain category similarity; Based on the text similarity and the category similarity, a category prediction is performed on the target image.

2. The method according to claim 1, characterized in that The calculating the similarity between the image description text and the target description text to obtain text similarity includes: Performing word segmentation processing on the image description text to obtain image description phrases, and performing word segmentation processing on the target description text to obtain target description phrases; removing stop words from the image description phrase to obtain an image description vocabulary, and removing stop words from the target description phrase to obtain a target description vocabulary; Calculating the vocabulary similarity of the image description vocabulary and the target description vocabulary to obtain initial text similarity; The initial text similarity is normalized to obtain the text similarity.

3. The method according to claim 2, characterized in that The calculating of vocabulary similarity between the image description vocabulary and the target description vocabulary to obtain initial text similarity includes: De-duplicating the image description vocabulary to obtain an image de-duplicated vocabulary, and de-duplicating the target description vocabulary to obtain a target de-duplicated vocabulary; Performing repeated word collection on the image deduplication words and the target deduplication words to obtain description intersection words; Merging the image description vocabulary and the target description vocabulary to obtain a description merged vocabulary; De-duplication of the description combined vocabulary is performed to obtain a description union vocabulary; Calculating the word length of the description intersection words to obtain a first word count, and calculating the word length of the description union words to obtain a second word count; A quotient operation is performed on the first vocabulary number and the second vocabulary number to obtain the initial text similarity.

4. The method according to claim 1, characterized in that The step of acquiring a sample image labeled with a reference image category and converting the sample image into text to obtain an image description text includes: Acquire an initial image, and perform image category labeling on the initial image to obtain the sample image; Dividing the sample image into blocks to obtain image blocks; Extracting features from the image block to obtain block features; Based on the image block features, a text description is performed on the image block to obtain an image block description text; The image description text is merged to obtain the image description text.

5. The method according to claim 4, characterized in that The step of performing a text description on the image block based on the image block feature to obtain the image block description text includes: Performing feature projection on the block features to obtain text features; The text feature is subjected to text decoding to obtain the image block description text.

6. The method according to any one of claims 1 to 5, characterized in that: The performing similarity calculation on the image feature and the reference image category to obtain category similarity includes: Encoding the reference image category to obtain a category identification vector; Performing cosine similarity calculation on the image feature and the category identification vector to obtain the initial category similarity; The initial category similarity is normalized to obtain the category similarity.

7. The method according to any one of claims 1 to 5, characterized in that: The step of predicting the category of the target image based on the text similarity and the category similarity includes: Performing similarity evaluation on the text similarity to obtain text similarity evaluation data; Performing similarity evaluation on the category similarity to obtain category similarity evaluation data; Merging the text similarity evaluation data and the category similarity evaluation data to obtain a target similarity; Based on the target similarity, the reference image category is screened to obtain an image prediction category.

8. An image category prediction device, characterized in that: The device comprises: A sample text description module is used to obtain a sample image marked with a reference image category, perform image-text conversion on the sample image, and obtain an image description text, wherein the reference image category is used to indicate the category of the sample image; A target text description module is used to obtain a target image, perform image-text conversion on the target image, and obtain a target description text; A text similarity calculation module is used to calculate the similarity between the image description text and the target description text to obtain text similarity; An image feature extraction module is used to extract features from the target image to obtain image features; A category similarity calculation module, used to perform similarity calculation on the image features and the reference image category to obtain category similarity; The image category prediction module is used to predict the category of the target image based on the text similarity and the category similarity.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the image category prediction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image category prediction method according to any one of claims 1 to 7 is implemented.

Images