Picture classification method and device and storage medium

Abstract

The invention discloses a picture classification method and device and a storage medium, and relates to the technical field of image processing. According to the method, the picture text is extracted through the OCR and the sentences are divided, the sentences are converted into the vectors by utilizing the pre-training vector generation model, the core semantic information associated with the category attributes in the text is fully mined by virtue of the semantic capture capability of the pre-training model, and the sentence text weight is determined by calculating the word frequency and the inverse document frequency after word segmentation. According to the method, the sentences corresponding to the important semantics are made to obtain higher weights, then the weights and the vector sequence are subjected to weighted average to generate the document vectors, the document vectors are made to accurately focus on the core semantics, the effectiveness of text representation is improved, and finally accurate classification is achieved through the classification model.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to image classification methods, devices and storage media. Background Technology

[0002] To achieve efficient management, retrieval, and utilization of massive amounts of images, image classification technology has become a research hotspot in the fields of computer vision and pattern recognition. Current image classification technologies mainly extract visual features such as color, texture, and shape from images using convolutional neural networks such as CNN, ResNet, and VGG, and then input these visual features into a classification model to determine the category.

[0003] However, in many image scenarios, such as archival images with text descriptions, e-commerce images with product descriptions, and government information disclosure images with annotations, the text information carried by the image is often directly related to its core category attributes. For example, the text of "financial voucher" images often includes information such as amount and voucher number, while the text of "product instruction manual" images focuses on product parameters and usage methods. The above-mentioned classification scheme based on visual features ignores the category association semantics contained in the text information of the image. When images of different categories have similar visual features but significantly different text information, it will lead to a decrease in classification accuracy.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide an image classification method, device, and storage medium, aiming to solve the technical problem of how to improve the accuracy of image classification.

[0006] To achieve the above objectives, this application proposes an image classification method, which includes: Extract the text from the image using the OCR engine, and segment the text into sentences to obtain a list of sentences; The sentence list is used as input to a pre-trained vector generation model, which converts the sentences in the sentence list into vectors to obtain a vector sequence. Each sentence in the sentence list is segmented into words, and the text weight of each sentence is determined based on the word frequency and inverse document frequency of each word in each sentence; The document vector is obtained by weighting the text weights and the vector sequence. The category label corresponding to the document vector is determined by a pre-trained classification model, which is used as the category to which the image belongs.

[0007] In one embodiment, the step of using the sentence list as input to a pre-trained vector generation model, and converting the sentences in the sentence list into vectors through the vector generation model to obtain a vector sequence includes: The sentence list is used as input to the vector generation model, and each sentence in the sentence list is converted into a token and attention mask through the input layer of the vector generation model; The transformer layer of the vector generation model uses the context features of each token as the hidden state of the token, and generates a vector sequence by weighting the attention mask corresponding to each token with the hidden state; Obtain the vector sequence output by the output layer of the vector generation model.

[0008] In one embodiment, before the step of obtaining a document vector by weighting the text weights and the vector sequence, and determining the category label corresponding to the document vector as the category to which the image belongs using a pre-trained classification model, the method further includes: Determine the character length of the sentence list, and determine the text density score based on the ratio of the character length to the total pixel area of ​​the image; The image processing model extracts visual feature vectors corresponding to each pixel region from the image, and determines the predicted text score corresponding to each pixel region based on the visual feature vectors. The predicted text score represents the probability that text exists in the pixel region. The difference between the sum of the predicted text scores corresponding to each pixel region and the text density score is determined, and the confidence score of the sentence list is determined based on the difference. When the confidence score is greater than or equal to a preset threshold, the following steps are performed: the document vector is obtained by weighting the text weights and the vector sequence, and the category label corresponding to the document vector is determined by a pre-trained classification model as the category to which the image belongs.

[0009] In one embodiment, after determining the difference between the sum of the predicted text scores corresponding to each of the pixel regions and the text density score, and determining the confidence score of the sentence list based on the difference, the method further includes: When the confidence score is less than the preset threshold, the region type of the pixel region is determined based on the visual feature vector corresponding to the pixel region. The visual weight corresponding to the pixel region is determined based on the region type; The sum of the visual weight of the pixel region to which each sentence belongs in the sentence list and the text weight of each sentence is used as the classification weight of the sentence. The classification weights are weighted and averaged with the vector sequence to obtain a document vector. The category label corresponding to the document vector is determined by a pre-trained classification model, which is used as the category to which the image belongs.

[0010] In one embodiment, after the step of obtaining a document vector by weighting the text weights and the vector sequence, and determining the category label corresponding to the document vector as the category to which the image belongs using a pre-trained classification model, the method further includes: Obtain the pixel coordinate information of the text output by the OCR engine in the image, and use the pixel coordinate information as the spatial feature of the text; A two-dimensional mesh is created based on a preset heatmap resolution; Based on the spatial features of the text, the classification weight corresponding to each sentence in the sentence list is mapped onto the two-dimensional grid to obtain a weight heatmap; The weighted heatmap is upsampled so that its resolution matches that of the image, and then rendered and overlaid on the image.

[0011] In one embodiment, the step of mapping the classification weight corresponding to each sentence in the sentence list to the two-dimensional grid based on the spatial features of the text to obtain a weight heatmap includes: Based on the spatial features of the text, the attention mask corresponding to each sentence is mapped onto the two-dimensional grid to determine the magnitude of the heat value of the two-dimensional grid; Based on the classification weight corresponding to each sentence, the heat value of the corresponding region of the two-dimensional grid is adjusted to obtain the weighted heat map, wherein the heat value is proportional to the classification weight.

[0012] In one embodiment, after the step of using the sentence list as input to a pre-trained vector generation model, and converting the sentences in the sentence list into vectors through the vector generation model to obtain a vector sequence, the method further includes: Based on a pre-defined ambiguous entity dictionary, target ambiguous words are matched in each of the sentences. Obtain other entity words within a preset text window of the target ambiguous word, and determine the domain to which the target ambiguous word belongs based on the semantic features of the other entity words; Obtain the knowledge graph of the domain to which the target ambiguous word belongs, and determine the knowledge vector of the target ambiguous word based on the knowledge graph; The vector corresponding to the target ambiguous word in the vector sequence is concatenated with the knowledge vector to obtain a new vector sequence.

[0013] In one embodiment, the step of obtaining a knowledge graph of the domain to which the target ambiguous word belongs, and determining the knowledge vector of the target ambiguous word based on the knowledge graph, includes: Obtain the knowledge graph of the domain to which the target ambiguous word belongs, and determine the target node corresponding to the target ambiguous word in the knowledge graph; Traverse the neighborhood graph of the target node, and determine the knowledge vector of the target ambiguous word based on the nodes and edges in the neighborhood graph.

[0014] In addition, to achieve the above objectives, this application also proposes an image classification device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image classification method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the image classification method described above.

[0016] This application provides an image classification method that extracts text from images using OCR and segments it into sentences. Then, a pre-trained vector generation model is used to convert the sentences into vectors. By leveraging the semantic capture capabilities of the pre-trained model, the core semantic information of the associated category attributes in the text is fully extracted. After word segmentation, word frequency and inverse document frequency are calculated to determine the text weight of each sentence, giving higher weights to sentences with important semantics. The weights are then weighted and averaged with the vector sequence to generate document vectors, which accurately focus on the core semantics, improving the effectiveness of text representation. Finally, accurate classification is achieved through a classification model. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the image classification method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the image classification method of this application; Figure 3This is a flowchart illustrating Embodiment 3 of the image classification method of this application; Figure 4 This is a flowchart illustrating Embodiment 4 of the image classification method of this application; Figure 5 This is a flowchart illustrating Embodiment 5 of the image classification method of this application; Figure 6 This is a simplified flowchart illustrating the image classification method in the embodiments of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the image classification method in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. It should be noted that all actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection laws and regulations of the country where the application is located, and with authorization from the owner of the corresponding device.

[0023] To achieve efficient management, retrieval, and utilization of massive amounts of images, image classification technology has become a research hotspot in the fields of computer vision and pattern recognition. Current image classification technologies mainly extract visual features such as color, texture, and shape from images using convolutional neural networks such as CNN, ResNet, and VGG, and then input these visual features into a classification model to determine the category.

[0024] However, in many image scenarios, such as archival images with text descriptions, e-commerce images with product descriptions, and government information disclosure images with annotations, the text information carried by the image is often directly related to its core category attributes. For example, the text of "financial voucher" images often includes information such as amount and voucher number, while the text of "product instruction manual" images focuses on product parameters and usage methods. The above-mentioned classification scheme based on visual features ignores the category association semantics contained in the text information of the image. When images of different categories have similar visual features but significantly different text information, it will lead to a decrease in classification accuracy.

[0025] In view of the above problems, this application proposes an image classification method. It extracts text from images using OCR and segments it into sentences. Then, it uses a pre-trained vector generation model to convert sentences into vectors. By leveraging the semantic capture capability of the pre-trained model, it fully mines the core semantic information of the associated category attributes in the text. After word segmentation, it calculates word frequency and inverse document frequency to determine the text weight of sentences, giving higher weight to sentences with important semantics. Then, it generates document vectors by weighted averaging of the weights and vector sequences, so that the document vectors accurately focus on the core semantics, improving the effectiveness of text representation. Finally, it achieves accurate classification through a classification model.

[0026] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or image classification system that can realize the above functions.

[0027] Based on this, the first embodiment of this application provides an image classification method, referring to... Figure 1 In this embodiment, the image classification method includes steps S10 to S40: Step S10: Obtain the text from the image in the OCR engine, and segment the text into sentences to obtain a list of sentences.

[0028] OCR (Optical Character Recognition) can identify text regions in an image and convert them into editable and searchable text data. Optionally, a series of preprocessing operations are performed on the image first, such as converting to grayscale, binarization, and noise reduction, to improve recognition accuracy. Next, the OCR engine locates the text regions in the image and performs character recognition, outputting all the recognized text as a continuous string. After obtaining the text recognized by the OCR engine, a rule-based or machine learning-based sentence segmentation module can be used to segment the continuous string output by the OCR engine into sentences, based on punctuation marks such as periods, question marks, line breaks, or semantic boundaries, to divide the continuous string into a list of sentences.

[0029] For example, if a machine learning-based sentence segmentation module is used to divide a continuous string into a list of sentences based on semantic boundaries, the continuous string can first be initially segmented into a list of lines based on newline characters, retaining the absolute position index and line number information of each character in the original string. Each line of text in the list is then further decomposed into smaller text units such as individual characters or words, generating a text sequence containing all text units, with each text unit accompanied by its character content, absolute position, and line number information.

[0030] The sentence segmentation module can then use a lightweight, pre-trained bidirectional context encoder, such as a mini-BERT (Bidirectional Encoder Representations from Transformers) or LSTM (Long Short-Term Memory), to process the entire text sequence. This bidirectional context encoder considers the contextual information of each text unit, generating a contextual semantic vector for each text unit in the text sequence to encode its semantic role within the overall text. Optionally, a set of symbolic and structural features can be computed in parallel for each text unit, primarily including punctuation features (e.g., whether it is a period, question mark, comma, etc.); lexical boundary features (by consulting a dictionary to determine whether the text unit is the beginning or end of a word); and positional and formatting features (e.g., whether it is at the beginning or end of a line, and its distance from the previous line). The contextual semantic vector of each unit is then concatenated with its symbolic and structural features to form a joint feature vector of the same length as the text sequence.

[0031] Subsequently, the sentence segmentation module can use a sequence labeling model to receive the entire joint feature vector. It analyzes the joint feature vector of each text unit using a sliding window or global inference approach, and, referencing the label prediction results of its neighboring text units, predicts a boundary label for each text unit in the text sequence. Boundary labels include: B-SEN (beginning of sentence), I-SEN (inside sentence), E-SEN (end of sentence), S-SEN (single character forming a sentence), etc. Scanning the text sequence with these boundary labels, when an E-SEN or S-SEN label is encountered, and the next label is B-SEN or the text sequence has reached its end, this is considered a complete semantic boundary. Then, based on the character content and absolute position of the corresponding text unit, all text units located between the two boundaries are concatenated into a sentence. Simultaneously, rules can be set within the sentence segmentation module to handle special cases such as consecutive punctuation marks and abbreviations, ensuring that the period in "Dr." is not misinterpreted as the end of a sentence.

[0032] Step S20: Use the sentence list as input to a pre-trained vector generation model, and use the vector generation model to convert the sentences in the sentence list into vectors to obtain a vector sequence.

[0033] A pre-trained vector generation model is a model pre-trained on a pre-defined corpus that can convert text into fixed-dimensional numerical vectors. These numerical vectors can capture the semantic information of the text. This vector generation model can be a context-aware model such as BERT.

[0034] Optionally, step S20 above includes steps S21 to S23: Step S21: Use the sentence list as the input of the vector generation model, and convert each sentence in the sentence list into tokens and an attention mask through the input layer of the vector generation model.

[0035] Exemplarily, the input layer of the vector generation model performs parallel word segmentation on each sentence in the sentence list, converting each sentence into a Token sequence. To enable batch calculation, among the Token sequences corresponding to all sentences, select the target Token sequence with the longest sequence length, and pad at the end of other Token sequences according to the sequence length of the target Token sequence, so that the Token sequences corresponding to all sentences are of equal length. At the same time, set the attention mask corresponding to the padding position of each sentence to 0, and the attention mask corresponding to the Token position to 1, to obtain the attention mask of each sentence.

[0036] It can be understood that when a deep learning framework performs matrix operations, it requires that all input tensors within a batch have the same size in each dimension. For text sequences, this means that for all sentences in a batch, the token sequences converted after word segmentation must be of equal length. However, the lengths of natural sentences are uneven. Therefore, through padding operations, meaningless padding characters can be added at the end of shorter token sequences to make them equal in length to the longest token sequence in the batch, so that they can be stacked into a regular three-dimensional tensor for one-time calculation. The above attention mask is used to cover those padding characters added for length alignment and without any semantics in self-attention calculation, preventing them from interfering with the representation learning of real text.

[0037] Optionally, there are also some stop words or low-information words in natural language such as "de", "le", etc. Based on the above-mentioned covering of padding characters without semantics, a semantic attention mask can be added and fused with the above attention mask to form a new attention mask.

[0038] Exemplarily, the input layer of the vector generation model loads predefined stop word tokens, matches and determines the target stop word tokens in the above token sequence, and generates a semantic attention mask of the same length as the token sequence for each sentence according to the length of the padded token sequence. Among them, the semantic attention mask corresponding to the target stop word token is set to 0, and the semantic attention mask corresponding to other tokens is set to 1. Perform an element-wise logical AND operation on the semantic attention mask and the above attention mask as the new attention mask for each sentence. In this new attention mask, the mask value is only 1 when the corresponding token is neither a padding character nor a stop word.

[0039] The above steps introduce semantic analysis into the mask generation stage, injecting stop word filtering into the forward propagation process of the model in the form of a mask. In the self-attention calculation stage of the Transformer core, the interaction between stop words and other words is directly weakened or blocked through semantic masks. This reduces noise interference from stop words when the model constructs the context features of each word, resulting in a higher signal-to-noise ratio and more accurate semantic meaning in the vector of the final generated sentence.

[0040] Step S22: The context features of each token are used as the hidden state of the token through the transformer layer of the vector generation model, and the attention mask corresponding to each token is weighted and averaged with the hidden state to generate a vector sequence.

[0041] Step S23: Obtain the vector sequence output by the output layer of the vector generation model.

[0042] For example, the token sequences corresponding to all the sentences are input into the transformer layer of the vector generation model. In the self-attention calculation of each transformer layer, the previously generated attention mask ensures that the tokens corresponding to the padding positions and stop word positions do not contribute to the information of other tokens. After calculation by multiple transformer layers, the representation of each token incorporates the contextual information of all other tokens in the entire sentence, forming the final hidden state. The attention mask corresponding to each sentence is normalized so that its sum is 1. Then, this attention mask is used as a weight to perform a weighted average of the hidden states corresponding to all non-padding tokens of the sentence, resulting in a vector sequence. Each row of the vector sequence represents the semantic representation of a sentence in the sentence list.

[0043] Step S30: Segment each sentence in the sentence list into words, and determine the text weight of each sentence based on the word frequency and inverse document frequency of each word in each sentence.

[0044] Term frequency (TF) measures how often a word appears in a single sentence, while inverse document frequency (IVF) measures the general importance of a word across a list of sentences. The product of the two can effectively assess how representative a word is of a particular sentence.

[0045] Optionally, the sentences in the sentence list are segmented, and stop words in the segmentation results are removed, retaining nouns, verbs, adjectives, and other content words to obtain a list of valid words. For the currently processed sentence, the frequency of each unique word in its list of valid words is determined, and this frequency is divided by the total number of valid words in the sentence to obtain the word frequency in the current sentence. Then, the list of valid words for all sentences is traversed to determine how many sentences each unique word appears in, obtaining the number of sentences containing that word. Finally, the inverse document frequency (IDF) formula is applied to calculate the IDF(word) = log(total number of sentences / (1 + number of sentences containing the word)) + 1.

[0046] For the sentence whose weight is to be calculated, iterate through the term frequency and inverse document frequency of all words in the sentence, multiply the two to obtain the term frequency and inverse document frequency weight of the word in the current sentence, and aggregate the term frequency and inverse document frequency weights of all valid words in the sentence to obtain the text weight of the sentence.

[0047] Step S40: The text weights are weighted and averaged with the vector sequence to obtain a document vector. The category label corresponding to the document vector is determined by a pre-trained classification model, which is used as the category to which the image belongs.

[0048] The text weights of the sentences above are weighted and averaged with the vectors corresponding to each sentence in the vector sequence to obtain the document vector of the image's text. This document vector is initially input into one or more fully connected layers of the classification model. These layers perform non-linear transformations and feature reorganization on the document vector, mapping it to a feature space more suitable for the classification task, thus obtaining document features. These document features are then fed into the output layer of the classification model. The number of neurons in the output layer equals the preset total number of class labels. This output layer calculates the score for each class label belonging to the document vector. A Softmax function is applied to all class label scores to convert them into a probability distribution, i.e., the predicted probability of each class label. The class label with the highest predicted probability is taken as the class label corresponding to the document vector.

[0049] It should be noted that before deploying the classification model, a batch of labeled image data can be used to train the model. For each image in this batch, all operations prior to document vector generation in steps S10 to S40 are performed, including OCR, sentence segmentation, sentence vectorization, and sentence weight calculation. Using the same algorithm and model parameters, a corresponding document vector is generated for each image, and each image is labeled with its true class label. The processed images are then input into the classification model, and the aforementioned forward propagation process is executed to obtain the predicted class probability of the document vector for each image. Subsequently, the loss function takes the predicted class probability and the true class label of the classification model as input parameters, and uses the cross-entropy loss function to calculate the difference between the two, obtaining a loss value to quantify the degree of error in the current classification model's prediction. Then, based on the gradient of this loss value relative to each parameter of the model, the value of each parameter is updated using the aforementioned gradient according to a preset optimization algorithm to reduce the loss value until the loss value is less than a preset threshold, completing the training and optimization of the classification model.

[0050] In this embodiment, by introducing a fusion semantic attention masking mechanism in the text vectorization stage, stop word noise can be actively suppressed during forward propagation, thereby improving the signal-to-noise ratio of the vector corresponding to the sentence. Furthermore, the weight calculation based on sentence-level word frequency and inverse document frequency can highlight sentences containing key and rare words, ensuring that the final generated document vector can highly condense and discriminately represent the text theme of the entire image, thus improving the accuracy of image classification.

[0051] Based on the first embodiment of this application, please refer to the second embodiment of this application. Figure 2 Before step S40, the image classification method further includes steps S50 to S80: Step S50: Determine the character length of the sentence list, and determine the text density score based on the ratio of the character length to the total pixel area of ​​the image.

[0052] For example, the sentence list is traversed to determine the total character length of all sentences in the sentence list. At the same time, the height and width of the image are obtained from the pixel data, the total pixel area is calculated, and the text density score is determined based on the ratio of the character length to the total pixel area. This text density score quantifies the amount of text recognized per unit area of ​​the image.

[0053] Step S60: Extract visual feature vectors corresponding to each pixel region from the image using a pre-trained image processing model, and determine the predicted text score corresponding to each pixel region based on the visual feature vectors. The predicted text score represents the probability that text exists in the pixel region.

[0054] The pre-trained image processing model can be a semantic segmentation or pixel-level classification model based on a convolutional neural network, such as U-Net or its variants. The image processing model inputs the entire image into its encoder, extracting deep visual feature maps through multiple convolutional layers and downsampling. These visual feature maps preserve spatial information, where the visual feature vector at each location corresponds to a pixel region in the image. The visual feature maps are then fed into the image processing model's decoder or a dedicated prediction head, which calculates a predicted text score for each pixel region in the visual feature maps, representing the likelihood that text exists in that pixel region, as determined by the image processing model.

[0055] For example, a visual feature vector is a high-dimensional numerical array generated by a pre-trained image processing model for each pixel region in an image. It does not directly represent color or brightness, but rather is an abstract encoding of the visual pattern of that pixel region learned by the image processing model through multiple convolutional neural networks. The visual feature vector captures combined information about the pixel region, such as specific edge combinations like horizontal and vertical strokes, texture patterns like regularly arranged spots, and shape context like the closed outline of characters. This combined information consists of discriminative features highly correlated with text presence, learned by the image processing model from massive amounts of image data during training.

[0056] After receiving an image, the image processing model feeds the image's pixel data, such as a 3D tensor representing height, width, and RGB color channels, into an initial convolutional layer. This initial convolutional layer scans the image, extracting basic visual features, such as simple edges and color gradients, and outputs a preliminary feature map. This preliminary feature map is then fed into an encoder module consisting of multiple convolutional layers and downsampling layers, such as pooling layers or strided convolutions. Convolution is used to fuse a wider range of contextual information, generating more complex and abstract feature representations; downsampling is used to compress the spatial size, expanding the receptive field of individual feature points, enabling them to capture semantic information from a broader area of ​​the image. After multiple convolutions and downsampling, the image is transformed into a deep visual feature map. Each point in the spatial dimension of the visual feature map corresponds to a pixel region in the image, and all the values ​​of that point in the channel dimension constitute the visual feature vector representing that pixel region. The visual feature map is then input into a prediction head, which acts as a miniature classifier, independently processing the visual feature vector of each pixel region in the visual feature map. Specifically, it performs a series of non-linear transformations and linear mappings on each visual feature vector through activation functions, ultimately compressing it into a single value. This value is then processed by the Sigmoid activation function and mapped to a value between 0 and 1, becoming the predicted text score for that pixel region. This score represents the probability that text exists in that pixel region, as determined by the image processing model based on the visual feature vector of that pixel region.

[0057] Step S70: Determine the difference between the sum of the predicted text scores corresponding to each pixel region and the text density score, and determine the confidence score of the sentence list based on the difference.

[0058] Understandably, the text density score, derived from the OCR recognition result itself, provides an objective, prior statistical feature based on the amount of output text. A high text density score for an image suggests it's likely a text-rich document such as a poster or instruction manual; conversely, a low score indicates sparse text, like a landscape image, or a severe OCR failure with numerous missed text entries. Predicting text scores bypasses the OCR engine, directly judging whether different areas of the image resemble text based on visual patterns such as texture, edges, and shapes from the raw pixel data. The image processing model can detect text areas that the OCR engine might miss, such as artistic fonts and blurred text. Simultaneously, for non-text areas that the OCR engine might misidentify as text, such as stripes or grids, the visual model typically assigns a low score.

[0059] A large difference between the sum of the predicted text scores for each pixel region and the text density score indicates that the image processing model identified a large number of text-like regions, but the OCR engine only read a small amount of text, or vice versa. This suggests that the confidence score of the sentence list recognized by the OCR engine is low. Conversely, a small difference between the sum of the predicted text scores for each pixel region and the text density score indicates that the visual text regions match the amount of text recognized by the OCR engine, resulting in a high confidence score for the sentence list.

[0060] Step S80: When the confidence score is greater than or equal to a preset threshold, perform the step of weighting the text weights and the vector sequence to obtain a document vector, and determining the category label corresponding to the document vector as the category to which the image belongs through a pre-trained classification model.

[0061] In this embodiment, when the image quality is poor, there is complex background interference, or the text itself has a special style, OCR recognition is prone to a large number of missed or false recognitions. This will directly lead to distortion of the subsequently generated document vectors, resulting in incorrect classification judgments. This embodiment introduces a cross-validation mechanism of text density score and visual predicted text score to intelligently evaluate the credibility of the OCR output results. The text-based deep classification process is only activated when the confidence score reaches a preset threshold, thereby avoiding blind reliance on low-quality, unreliable OCR results at the system level and significantly reducing the overall classification error rate caused by front-end recognition errors.

[0062] Based on the second embodiment of this application, please refer to the third embodiment of this application. Figure 3 After step S70, the image classification method further includes steps S90 to S120: Step S90: When the confidence score is less than the preset threshold, determine the region type of the pixel region based on the visual feature vector corresponding to the pixel region.

[0063] Step S100: Determine the visual weight corresponding to the pixel region based on the region type.

[0064] When the confidence scores of the sentence list are insufficient, the image processing model's prediction head independently processes the visual feature vector of each pixel region. This prediction head is trained to map visual feature vectors to predefined region types, such as "clear text," "artistic font," "background texture," "icon," "image region," and "decorative border." The prediction head calculates the probability of each pixel region belonging to each region type and selects the region type with the highest probability as the region type for that pixel region.

[0065] Image processing models internally pre-define or learn a region type-visual weight mapping table, which defines the contribution of each region type to the semantic importance of a sentence. The logic is that regions directly related to the text's semantics or potentially carrying important text should be assigned higher weights, and vice versa. For example, "clear body text" regions correspond to high visual weights because their content is highly credible. "Icons" regions correspond to medium visual weights because they may contain key information, while regions such as "background textures" and "decorative borders" correspond to low or zero visual weights.

[0066] Step S110: The sum of the visual weights of the pixel regions to which each sentence belongs in the sentence list and the text weights of each sentence is used as the classification weight of the sentence.

[0067] For each sentence in the sentence list, based on the coordinates of the text box corresponding to the sentence during OCR engine recognition, determine all pixel regions covered by that text box, i.e., the corresponding pixel region. Calculate the average or maximum visual weight of that pixel region as the overall visual weight of the sentence. Add the sentence's text weight to its visual weight to obtain the sentence's classification weight.

[0068] Step S120: The classification weights are weighted and averaged with the vector sequence to obtain a document vector. The category label corresponding to the document vector is determined by a pre-trained classification model, which is used as the category to which the image belongs.

[0069] The execution process of step S120 is similar to step S40 in the first embodiment, but the weights used are different. In this embodiment, the classification weight list is first normalized. Then, the vector of each sentence in the vector sequence is multiplied by its corresponding classification weight, and then a weighted average is performed to generate the final document vector. This document vector considers not only the statistical importance of the text but also incorporates visual semantic importance correction. Finally, this document vector is input into a pre-trained classification model to determine its category label, which is used as the category to which the image belongs.

[0070] In this embodiment, when the confidence score of the sentence list is insufficient, visual context information is introduced to dynamically correct and fuse the text weights, thereby generating new classification weights. This enables the image classification method to effectively process images with complex layouts, special fonts, or background interference, avoiding the reduction in classification accuracy caused by poor OCR recognition quality. This significantly enhances the applicability and practicality of the technical solution in diverse and non-ideal real-world scenarios.

[0071] Based on the above embodiments of this application, in the fourth embodiment of this application, please refer to... Figure 4 After step S40, the image classification method further includes steps S130 to S160: Step S130: Obtain the pixel coordinate information of the text output by the OCR engine in the image, and use the pixel coordinate information as the spatial feature of the text.

[0072] For example, the pixel coordinate information of each text unit in the image is parsed from the OCR output. This pixel coordinate information defines the layout and geometric relationship of the text in the two-dimensional image space, constituting the spatial features of the text.

[0073] Step S140: Create a two-dimensional mesh based on the preset heatmap resolution.

[0074] Step S150: Based on the spatial features of the text, map the classification weight corresponding to each sentence in the sentence list onto the two-dimensional grid to obtain a weight heatmap.

[0075] A weighted heatmap is an intuitive, spatially distributed two-dimensional visualization of text within an image, generated from the pixel coordinates of the text. Each region in the weighted heatmap, corresponding to a cell in a two-dimensional grid, is assigned a color or heat value, which is proportional to the sum of the classification weights of the sentences mapped to the corresponding pixel region of the image.

[0076] For example, a two-dimensional grid of a corresponding size is created based on a preset heatmap resolution that is lower than the image resolution. Each cell of this grid, relative to each pixel, is used to accumulate and represent weight information within its corresponding image region. For each sentence in the sentence list, the range of cells in the two-dimensional grid that the sentence covers or primarily influences is determined based on its corresponding spatial features. Then, the classification weight value of the sentence is added to the current value of the target cell within the aforementioned cell range according to certain rules, such as uniform distribution or distribution based on the proportion of the overlap area between the cell and the text box. After traversing all sentences, the value of each grid cell reflects the importance weight of the text at that spatial location for the final classification decision, thus forming a weighted heatmap.

[0077] Optionally, step S150 above includes steps S151 to S152: Step S151: Based on the spatial features of the text, map the attention mask corresponding to each sentence onto the two-dimensional grid to determine the magnitude of the heat value of the two-dimensional grid.

[0078] Step S152: Adjust the heat value of the corresponding region of the two-dimensional grid according to the classification weight corresponding to each sentence to obtain the weight heat map, wherein the heat value is proportional to the classification weight.

[0079] For example, for each sentence in the sentence list, firstly, the attention mask and corresponding pixel coordinate information of that sentence are obtained. Based on the pixel coordinate information of the sentence, the range of cells covered by the sentence on the two-dimensional grid is calculated. Next, the one-dimensional attention mask sequence of the sentence is mapped to the range of grid cells calculated above, according to the relative positions of each word in the sentence. Specifically, a heat value is calculated for each covered grid cell: if the text position corresponding to the cell is marked as "1" in the attention mask, the heat value is 1; if the text position corresponding to the cell is marked as "0" in the attention mask, the heat value is 0. All sentences are traversed, and the heat values ​​of all grid cells covered by each sentence are accumulated.

[0080] Then, the sentence list is traversed again. For each sentence, the affected 2D grid cell region is determined based on its pixel coordinates. The sentence's classification weight is then used as a modulation factor, applied to the heatmap values ​​of each cell within this region, resulting in a weighted heatmap. The adjustment method can be weighted summation, such as adding the product of the sentence's classification weight and its heatmap value in the current cell to the cell's current value; or it can be taking the maximum value, such as updating the cell value using the larger of the sentence's classification weight and the cell's current value. After traversing all sentences, the final heatmap value of each grid cell is the combined result of modulating the cell's previous heatmap value based on each sentence's own classification weight.

[0081] Step S160: Upsample the weighted heatmap so that its resolution matches that of the image, and then render and overlay the weighted heatmap onto the image.

[0082] The weighted heatmap is upsampled to the same resolution as the original image using image interpolation methods such as bilinear interpolation. Then, the heat values ​​in the upsampled weighted heatmap are converted into a color image through a semi-transparent color mapping. This color image is then rendered and overlaid on the original image with a preset transparency to generate the final visualization result.

[0083] The above steps first use attention masks to finely characterize the true distribution of text at the word level, revealing which parts of a sentence are actually focused on by the model. Then, classification weights are introduced to modulate global importance at the sentence level, ensuring that the highlighted areas of the final heatmap are semantically perfectly aligned with the core criteria for classification decisions. The weighted heatmap generated through these steps not only shows where the text is, but more accurately shows which text is more important for the model's classification judgment. It provides developers with more powerful debugging tools; for example, they can observe whether the highlighted areas fall on meaningful content words or irrelevant function words, thereby deeply evaluating the effectiveness of the vector generation model and weight calculation module.

[0084] Based on the above embodiments of this application, please refer to the fifth embodiment of this application. Figure 5 After step S20, the image classification method further includes steps S170~S200: Step S170: Based on a preset ambiguous entity dictionary, match the target ambiguous word in each of the sentences.

[0085] The algorithm iterates through each sentence in the sentence list. For each word in the sentence, it queries a pre-defined ambiguous entity dictionary to identify the target ambiguous word in the sentence. The ambiguous entity dictionary is a structured vocabulary used to quickly identify ambiguous words in text that require deep semantic resolution. For example, the word "apple" can have semantic meanings including fruit, technology company, movie / brand, etc., and such ambiguous words can interfere with image classification.

[0086] Step S180: Obtain other entity words within a preset text window of the target ambiguous word, and determine the domain to which the target ambiguous word belongs based on the semantic features of the other entity words.

[0087] For each target ambiguous word identified in a sentence, all other entity words within a predefined text window centered on the target ambiguous word are extracted as a context fragment. The semantic features of all other entity words in this context fragment are analyzed, and their semantic features are extracted. Based on these semantic features, rule matching or a lightweight classifier is used to infer the most likely domain of the current target ambiguous word in this specific context. For example, in "buying apples and bananas," the domain of "apple" is identified as "fruit"; in "Apple releases a new phone," its domain is identified as "technology company."

[0088] Step S190: Obtain the knowledge graph of the domain to which the target ambiguous word belongs, and determine the knowledge vector of the target ambiguous word based on the knowledge graph.

[0089] Based on the domain of the target ambiguous word, a knowledge graph corresponding to that domain is obtained or loaded. The node representing the target ambiguous word is queried within this knowledge graph, and its associated attributes, relationships, and other structured information are retrieved. Finally, this structured information is encoded into a fixed-dimensional knowledge vector using a knowledge encoder, such as a graph embedding model like TransE, or a simple graph neural network. This knowledge vector deeply represents the unique and accurate semantic knowledge of the word within its specific domain.

[0090] Optionally, step S190 above includes steps S191 to S192: Step S191: Obtain the knowledge graph of the domain to which the target ambiguous word belongs, and determine the target node corresponding to the target ambiguous word in the knowledge graph.

[0091] For example, first, the knowledge graph corresponding to the domain of the target ambiguous word is loaded, and the text content of the target ambiguous word is matched with the similarity of the names, aliases, and other attributes of all nodes in the knowledge graph. The node with the highest similarity is then selected from all nodes in the knowledge graph and identified as the target node.

[0092] Step S192: Traverse the neighborhood graph of the target node, and determine the knowledge vector of the target ambiguous word based on the nodes and edges in the neighborhood graph.

[0093] Centered on the target node, a bounded traversal is performed within the knowledge graph, for example, sampling nodes and edges within a 1-hop or 2-hop range to obtain a local neighborhood graph. This neighborhood graph contains the target node, its directly connected neighboring nodes, and the edges connecting them. The semantic attributes of the target node, its neighboring nodes, and the relational attributes represented by the edges connecting them are input as a lightweight graph neural network or relational path encoder. The graph neural network or relational path encoder learns to aggregate these discrete nodes and edges into a continuous, fixed-dimensional vector representation. This results in a knowledge vector that not only includes the concept of the target node itself but also deeply integrates its role, attributes, and relational network within the specific knowledge graph.

[0094] Step S200: Concatenate the vector corresponding to the target ambiguous word in the vector sequence with the knowledge vector to obtain a new vector sequence.

[0095] For sentences containing target ambiguities, the original vector of the sentence is concatenated with the knowledge vector corresponding to the target ambiguity to form a new vector sequence. If a sentence contains multiple target ambiguities, multiple knowledge vectors can be concatenated sequentially, or the average of the knowledge vectors can be taken before concatenation. The vector representation of the sentence in the vector sequence is then updated.

[0096] This embodiment actively identifies target ambiguous words in sentences using a pre-defined ambiguous entity dictionary and dynamically determines their domain in the current context by utilizing the semantic features of other entity words within a pre-defined text window. Then, it extracts and encodes their structured association information from the knowledge graph of the corresponding domain, generating accurate knowledge vectors, which are then fused with the original sentence vectors. This achieves a complete closed loop from potential ambiguity to contextual disambiguation to knowledge enhancement, ensuring accurate characterization of key entity semantics. The resulting new vector sequence has a clearer semantic space distribution, effectively distinguishing similar expressions across different domains, laying a solid foundation for the subsequent generation of high-quality document vectors.

[0097] For example, to help understand the implementation process of the image classification method obtained by combining the above embodiments, please refer to... Figure 6 , Figure 6 A simplified flowchart of an image classification method is provided, specifically: First, OCR recognition is performed on the input image to obtain the original text string. Then, a sentence segmentation module transforms it into a structured list of sentences. Next, the sentence list is converted into a vector sequence, target ambiguous words in the sentences are identified, and the corresponding domain knowledge graph is queried to generate knowledge vectors that are then concatenated with the vectors, resulting in a new vector sequence with more accurate semantics. Text weights are calculated for each sentence using term frequency and inverse document frequency (IVF) to quantify its importance based on term frequency statistics. A text density score is calculated, and the difference between the text density score and the predicted text score is used to calculate the sentence's confidence score. When the confidence score is greater than or equal to a preset threshold, the text recognition result is considered reliable. The text weights are then directly weighted and averaged with the vector sequence to generate a document vector and determine the category label. When the confidence score is less than the preset threshold, text recognition may have issues. A pre-trained image model is used to analyze the image, determine the region type of each pixel, and assign visual weights accordingly. The visual weights of the region to which the sentence belongs are added to its original text weights to obtain more robust classification weights that incorporate visual context. Finally, the classification weights are weighted and averaged with the vector sequence to generate more reliable document vectors and determine the category label. The image type is then determined based on the category label. Afterward, the classification weights are mapped onto a two-dimensional grid using the pixel coordinates of the text, generating a weight heatmap. This heatmap is then upsampled and rendered overlaid on the original image. This visually demonstrates to the user which text regions in the image the model uses to make classification decisions.

[0098] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the image classification method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0099] This application provides an image classification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the image classification method in Embodiment 1 above.

[0100] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an image classification device suitable for implementing embodiments of this application. The image classification device in these embodiments may include, but is not limited to, mobile terminals such as laptops and tablets (PADs), and fixed terminals such as desktop computers. Figure 7 The image classification device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0101] like Figure 7As shown, the image classification device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the image classification device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the image classification device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show image classification devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0102] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0103] The image classification device provided in this application, employing the image classification method described in the above embodiments, can solve the technical problem of how to improve the accuracy of image classification. Compared with the prior art, the beneficial effects of the image classification device provided in this application are the same as those of the image classification method provided in the above embodiments, and other technical features of this image classification device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0104] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0105] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0106] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the image classification method in the above embodiments.

[0107] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0108] The aforementioned computer-readable storage medium may be included in the image classification device; or it may exist independently and not be assembled into the image classification device.

[0109] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the image classification device, enable the image classification device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, or as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0111] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0112] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image classification method, thereby solving the technical problem of how to improve the accuracy of image classification. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the image classification method provided in the above embodiments, and will not be repeated here.

[0113] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image classification method described above.

[0114] The computer program product provided in this application can solve the technical problem of how to improve the accuracy of image classification. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the image classification method provided in the above embodiments, and will not be repeated here.

[0115] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An image classification method, characterized in that, The image classification method includes: Extract the text from the image using the OCR engine, and segment the text into sentences to obtain a list of sentences; The sentence list is used as input to a pre-trained vector generation model, which converts the sentences in the sentence list into vectors to obtain a vector sequence. Each sentence in the sentence list is segmented into words, and the text weight of each sentence is determined based on the word frequency and inverse document frequency of each word in each sentence; The document vector is obtained by weighting the text weights and the vector sequence. The category label corresponding to the document vector is determined by a pre-trained classification model, which is used as the category to which the image belongs.

2. The image classification method as described in claim 1, characterized in that, The step of using the sentence list as input to a pre-trained vector generation model, and converting the sentences in the sentence list into vectors through the vector generation model to obtain a vector sequence includes: The sentence list is used as input to the vector generation model, and each sentence in the sentence list is converted into a token and attention mask through the input layer of the vector generation model; The transformer layer of the vector generation model uses the context features of each token as the hidden state of the token, and generates a vector sequence by weighting the attention mask corresponding to each token with the hidden state; Obtain the vector sequence output by the output layer of the vector generation model.

3. The image classification method as described in claim 1, characterized in that, Before the step of obtaining a document vector by weighting the text weights and the vector sequence, and determining the category label corresponding to the document vector as the category to which the image belongs using a pre-trained classification model, the method further includes: Determine the character length of the sentence list, and determine the text density score based on the ratio of the character length to the total pixel area of ​​the image; The image processing model extracts visual feature vectors corresponding to each pixel region from the image, and determines the predicted text score corresponding to each pixel region based on the visual feature vectors. The predicted text score represents the probability that text exists in the pixel region. The difference between the sum of the predicted text scores corresponding to each pixel region and the text density score is determined, and the confidence score of the sentence list is determined based on the difference. When the confidence score is greater than or equal to a preset threshold, the following steps are performed: the document vector is obtained by weighting the text weights and the vector sequence, and the category label corresponding to the document vector is determined by a pre-trained classification model as the category to which the image belongs.

4. The image classification method as described in claim 3, characterized in that, After determining the difference between the sum of the predicted text scores corresponding to each of the pixel regions and the text density score, and determining the confidence score of the sentence list based on the difference, the method further includes: When the confidence score is less than the preset threshold, the region type of the pixel region is determined based on the visual feature vector corresponding to the pixel region. The visual weight corresponding to the pixel region is determined based on the region type; The sum of the visual weight of the pixel region to which each sentence belongs in the sentence list and the text weight of each sentence is used as the classification weight of the sentence. The classification weights are weighted and averaged with the vector sequence to obtain a document vector. The category label corresponding to the document vector is determined by a pre-trained classification model, which is used as the category to which the image belongs.

5. The image classification method as described in claim 4, characterized in that, After the step of obtaining a document vector by weighting the text weights and the vector sequence, and determining the category label corresponding to the document vector as the category to which the image belongs through a pre-trained classification model, the method further includes: Obtain the pixel coordinate information of the text output by the OCR engine in the image, and use the pixel coordinate information as the spatial feature of the text; A two-dimensional mesh is created based on a preset heatmap resolution; Based on the spatial features of the text, the classification weight corresponding to each sentence in the sentence list is mapped onto the two-dimensional grid to obtain a weight heatmap; The weighted heatmap is upsampled so that its resolution matches that of the image, and then rendered and overlaid on the image.

6. The image classification method as described in claim 5, characterized in that, The step of mapping the classification weight corresponding to each sentence in the sentence list to the two-dimensional grid based on the spatial features of the text to obtain a weight heatmap includes: Based on the spatial features of the text, the attention mask corresponding to each sentence is mapped onto the two-dimensional grid to determine the magnitude of the heat value of the two-dimensional grid; Based on the classification weight corresponding to each sentence, the heat value of the corresponding region of the two-dimensional grid is adjusted to obtain the weighted heat map, wherein the heat value is proportional to the classification weight.

7. The image classification method as described in claim 1, characterized in that, After the step of using the sentence list as input to a pre-trained vector generation model, and converting the sentences in the sentence list into vectors to obtain a vector sequence, the method further includes: Based on a pre-defined ambiguous entity dictionary, target ambiguous words are matched in each of the sentences. Obtain other entity words within a preset text window of the target ambiguous word, and determine the domain to which the target ambiguous word belongs based on the semantic features of the other entity words; Obtain the knowledge graph of the domain to which the target ambiguous word belongs, and determine the knowledge vector of the target ambiguous word based on the knowledge graph; The vector corresponding to the target ambiguous word in the vector sequence is concatenated with the knowledge vector to obtain a new vector sequence.

8. The image classification method as described in claim 7, characterized in that, The step of obtaining the knowledge graph of the domain to which the target ambiguous word belongs, and determining the knowledge vector of the target ambiguous word based on the knowledge graph, includes: Obtain the knowledge graph of the domain to which the target ambiguous word belongs, and determine the target node corresponding to the target ambiguous word in the knowledge graph; Traverse the neighborhood graph of the target node, and determine the knowledge vector of the target ambiguous word based on the nodes and edges in the neighborhood graph.

9. An image classification device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image classification method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the image classification method as described in any one of claims 1 to 8.

Images