Image classification method, image classification apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide an image classification method, an image classification device, an electronic device and a storage medium, and belong to the technical field of artificial intelligence. The method comprises: obtaining a target image to be processed; performing text recognition on the target image to obtain original text data; performing structured processing on the original text data according to a preset algorithm to obtain line text data; performing classification processing on the line text data through a preset text classification model to obtain first classification data; performing classification processing on the original text data through a preset regular matching mode to obtain second classification data; and obtaining target classification data according to the first classification data and the second classification data. The embodiments of the present application can improve the accuracy of image classification.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image classification method, an image classification device, an electronic device, and a storage medium. Background Technology

[0002] In the medical field, it is often necessary to classify different medical images. However, the current classification process often relies on manual classification, which is highly subjective and affects the accuracy of image classification. Therefore, how to improve the accuracy of image classification has become an urgent technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to provide an image classification method, an image classification device, an electronic device, and a storage medium, with the aim of improving the accuracy of image classification.

[0004] To achieve the above objectives, a first aspect of this application proposes an image classification method, the method comprising:

[0005] Obtain the target image to be processed;

[0006] Perform text recognition on the target image to obtain the original text data;

[0007] The original text data is structured according to a preset algorithm to obtain line text data;

[0008] The line text data is classified using a preset text classification model to obtain the first category data;

[0009] The original text data is classified using a preset regular expression matching method to obtain second-classified data;

[0010] Based on the first classification data and the second classification data, the target classification data is obtained.

[0011] In some embodiments, the step of performing structured processing on the original text data according to a preset algorithm to obtain line text data includes:

[0012] The original text data is sorted using a preset sorting algorithm to obtain an initial text sequence;

[0013] The initial text sequence is fitted using the least squares method to obtain the line text slope values.

[0014] The original text data is processed by line structuring based on the line text slope value to obtain the line text data.

[0015] In some embodiments, the text classification model includes convolutional networks and ensemble networks. The step of classifying the line text data using a preset text classification model to obtain first-classified data includes:

[0016] The line text data is embedded to obtain a line text embedding vector;

[0017] The convolutional network is used to extract features from the line text embedding vector to obtain the line text representation vector.

[0018] The combined text representation vectors are processed by the combined network to obtain a fused text feature vector;

[0019] The line text representation vector is subjected to max pooling to obtain the line text pooling vector;

[0020] The fused line text feature vector and the line text pooling vector are concatenated to obtain the target text representation vector.

[0021] The target text representation vector is classified using a preset function to obtain the first classification data.

[0022] In some embodiments, the convolutional network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a pooling layer. The step of extracting features from the line text embedding vector through the convolutional network to obtain the line text representation vector includes:

[0023] The first convolutional layer performs convolution processing on the line text embedding vector to obtain a first convolutional vector, and the pooling layer performs max pooling processing on the first convolutional vector to obtain a first pooling vector.

[0024] The second convolutional layer performs convolution processing on the line text embedding vector to obtain a second convolutional vector, and the pooling layer performs max pooling processing on the second convolutional vector to obtain a second pooling vector.

[0025] The line text embedding vector is convolved by the three convolutional layers to obtain the third convolutional vector, and then the third convolutional vector is max-pooled by the pooling layer to obtain the third pooling vector.

[0026] The first pooling vector, the second pooling vector, and the third pooling vector are concatenated to obtain the line text representation vector.

[0027] In some embodiments, the step of combining the line text representation vectors through the combined network to obtain a fused line text feature vector includes:

[0028] The classification function of the combined network is used to calculate the weights of the line text representation vectors to obtain the classification weights of each line text representation vector.

[0029] The fused text feature vector is obtained by weighting the line text representation vector according to the classification weight.

[0030] In some embodiments, the step of classifying the target text representation vector using a preset function to obtain the first classification data includes:

[0031] The classification probability of the target text representation vector is calculated by using the preset function and preset text category labels to obtain the classification probability value of each text category label;

[0032] The first classification data is obtained based on the classification probability value.

[0033] In some embodiments, the step of obtaining target classification data based on the first classification data and the second classification data includes:

[0034] The first category data and the second category data are compared and analyzed to obtain the analysis results;

[0035] If the analysis result shows that the first category data and the second category data are the same, then the first category data or the second category data shall be used as the target category data.

[0036] If the analysis result shows that the first category data and the second category data are different, then the priority of the first category data and the second category data is obtained, and the first category data or the second category data is used as the target category data according to the priority.

[0037] To achieve the above objectives, a second aspect of this application provides an image classification apparatus, the apparatus comprising:

[0038] The image acquisition module is used to acquire the target image to be processed;

[0039] The text recognition module is used to perform text recognition on the target image to obtain the original text data;

[0040] The structured processing module is used to perform structured processing on the original text data according to a preset algorithm to obtain line text data;

[0041] The first classification module is used to classify the line text data using a preset text classification model to obtain the first classification data;

[0042] The second classification module is used to classify the original text data using a preset regular expression matching method to obtain second classification data;

[0043] The comparison module is used to obtain target classification data based on the first classification data and the second classification data.

[0044] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the method described in the first aspect above.

[0045] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the method described in the first aspect.

[0046] The image classification method, device, electronic equipment, and storage medium proposed in this application acquire a target image to be processed; perform text recognition on the target image to obtain raw text data, which can conveniently obtain the semantic content information of the target image; further, perform structured processing on the raw text data according to a preset algorithm to obtain line text data, which can conveniently obtain the structural layout features of the target image. Classify the line text data using a preset text classification model to obtain first-class data, which improves classification accuracy. Simultaneously, classify the raw text data using a preset regular expression matching method to obtain second-class data, which ensures the rationality of the classification. Finally, based on the first-class data and the second-class data, obtain the target classification data. This method combines regular expression matching and model classification to determine the category of the target image, thereby effectively improving the accuracy of image classification. Attached Figure Description

[0047] Figure 1 This is a flowchart of the image classification method provided in the embodiments of this application;

[0048] Figure 2 yes Figure 1 The flowchart of step S103 in the process;

[0049] Figure 3 yes Figure 1 The flowchart of step S104 in the process;

[0050] Figure 4 yes Figure 3The flowchart of step S302 in the document;

[0051] Figure 5 yes Figure 3 The flowchart of step S303 in the process;

[0052] Figure 6 yes Figure 3 The flowchart of step S306 in the process;

[0053] Figure 7 yes Figure 1 The flowchart of step S106 in the process;

[0054] Figure 8 This is a schematic diagram of the structure of the image classification device provided in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] First, let's analyze some of the terms used in this application:

[0060] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0061] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0062] Information Extraction (NER) is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0063] Magnetic Resonance Imaging (MRI): MRI, also known as magnetic resonance imaging, is a physical phenomenon widely used as an analytical tool in fields such as physics, chemistry, and biology. MRI works by applying radio frequency pulses of a specific frequency to the human body within a static magnetic field, exciting hydrogen protons in the body and causing a magnetic resonance phenomenon.

[0064] Medical imaging: Medical imaging has a variety of imaging modalities, such as magnetic resonance (MR), computed tomography (CT), PET, ultrasound (US) imaging, etc.

[0065] CT (Computed Tomography): This is a type of computed tomography scan that uses precisely collimated X-ray beams, gamma rays, ultrasound, etc., along with highly sensitive detectors to scan a specific part of the human body one section after another. It features fast scanning time and clear images and can be used to examine a variety of diseases. Depending on the type of radiation used, it can be divided into X-ray CT (X-CT) and gamma-ray CT (γ-CT), etc.

[0066] Medical imaging refers to the techniques and processes used to non-invasively acquire images of internal tissues of the human body or a part of the human body for medical treatment or research. It encompasses two relatively independent research directions: medical imaging systems and medical image processing. The former refers to the image formation process, including research on imaging mechanisms, imaging equipment, and imaging system analysis; the latter refers to further processing of acquired images, with purposes such as restoring unclear images, highlighting certain features, or performing pattern classification.

[0067] Embedding: An embedding is a vector representation that uses a low-dimensional vector to represent an object. This object can be a word, a product, a movie, etc. The property of an embedding vector is that vectors with close proximity correspond to objects with similar meanings. For example, the embeddings of "Avengers" and "Iron Man" are very close, but the embeddings of "Avengers" and "Gone with the Wind" are farther apart. Essentially, embedding is a mapping from semantic space to vector space, while preserving the semantic relationships of the original samples in the vector space as much as possible. For instance, two semantically similar words are also relatively close in the vector space. Embedding can encode objects using low-dimensional vectors while retaining their meaning. It is commonly used in machine learning. In the process of building machine learning models, objects are encoded into low-dimensional dense vectors before being fed into a deep neural network (DNN) to improve efficiency.

[0068] Pooling is an important concept in convolutional neural networks; it is essentially a form of downsampling. There are various non-linear pooling functions, with max pooling being the most common. Max pooling divides the input image into several rectangular regions and outputs the maximum value for each sub-region.

[0069] Regularization refers to the process in linear algebra where ill-posed problems are typically defined by a set of linear algebraic equations, often stemming from ill-posed inverse problems with large condition numbers. Large condition numbers mean that rounding errors or other errors can severely impact the solution.

[0070] In the medical field, it is often necessary to classify different medical images. However, the current classification process often relies on manual classification, which is highly subjective and affects the accuracy of image classification. Therefore, how to improve the accuracy of image classification has become an urgent technical problem to be solved.

[0071] Based on this, embodiments of this application provide an image classification method, an image classification device, an electronic device, and a storage medium, aiming to improve the accuracy of image classification.

[0072] The image classification method, image classification device, electronic device and storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the image classification method in the embodiments of this application is described.

[0073] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0074] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0075] The image classification method provided in this application relates to the field of artificial intelligence technology. The image classification method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the image classification method, but is not limited to the above forms.

[0076] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0077] Figure 1 This is an optional flowchart of the image classification method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0078] Step S101: Obtain the target image to be processed;

[0079] Step S102: Perform text recognition on the target image to obtain the original text data;

[0080] Step S103: Perform structured processing on the original text data according to the preset algorithm to obtain line text data;

[0081] Step S104: Classify the line text data using a preset text classification model to obtain the first category data;

[0082] Step S105: Classify the original text data using a preset regular expression matching method to obtain the second category data;

[0083] Step S106: Obtain target classification data based on the first classification data and the second classification data.

[0084] Steps S101 to S106 of this embodiment involve acquiring a target image to be processed; performing text recognition on the target image to obtain original text data, which facilitates the acquisition of semantic content information of the target image; further, performing structured processing on the original text data according to a preset algorithm to obtain line text data, which facilitates the acquisition of structural layout features of the target image. The line text data is then classified using a preset text classification model to obtain first-classification data, which improves classification accuracy. Simultaneously, the original text data is classified using a preset regular expression matching method to obtain second-classification data, ensuring reasonable classification. Finally, based on the first-classification data and the second-classification data, target classification data is obtained. This approach combines regular expression matching and model classification to determine the category of the target image, thereby effectively improving the accuracy of image classification.

[0085] In step S101 of some embodiments, the target image to be processed is a three-dimensional image, which may be obtained through CT or MRI. The target image to be processed may include the target object's medical records, surgical records, medical condition records, examination records, etc., and the target object may be a patient or other population.

[0086] In some medical applications, the target image mentioned above can be a medical image, containing an object classified as a lesion, that is, a part of the body where a disease has occurred. Medical images refer to internal tissues obtained non-invasively for medical treatment or research, such as CT, MRI, ultrasound, X-ray images, and images generated by medical instruments using optical imaging.

[0087] In step S102 of some embodiments, the target image is preprocessed using an OCR text recognition tool or the like. This image preprocessing process includes image binarization, image denoising, tilt correction, and character segmentation, etc. For example, in the image binarization process, if the foreground information is defined as black and the background information as white, the color target image can be converted into a grayscale image. After preprocessing the target image, text features are extracted from the preprocessed target image. The extracted text features are compared and recognized with the reference text in a preset text character library to obtain initial text data. Finally, the initial text data is corrected according to preset grammar rules or contextual relationships to obtain the original text data.

[0088] When performing text recognition on a target image, the image is often rotated. After rotation, the coordinates of the elements in the target image also change. This phenomenon affects the recognition of the line content information of the target image. Therefore, after performing overall document recognition on the target image, it is also necessary to perform structured processing on the recognized raw text data to improve the recognition accuracy.

[0089] Please see Figure 2 In some embodiments, step S103 may include, but is not limited to, steps S201 to S203:

[0090] Step S201: Sort the original text data using a preset sorting algorithm to obtain an initial text sequence;

[0091] Step S202: Fit the initial text sequence using the least squares method to obtain the line text slope value;

[0092] Step S203: Perform line structuring processing on the original text data based on the line text slope value to obtain line text data.

[0093] In step S201 of some embodiments, the preset sorting algorithm can be a row-forming algorithm, which restores the original text data in the form of a list. The row-forming algorithm divides the entire original text data into multiple adjacent rectangles. Each rectangle includes 10 parameters: the horizontal and vertical coordinates of its four vertices, the OCR recognition confidence score, and the text fragments recognized by the OCR text recognition tool. When calculating the vertex coordinates of each rectangle, the first character element at the top left corner of the original text data can be used as the coordinate zero point. The horizontal coordinate of the original text data is x, and the vertical coordinate is y. This method conveniently splits the original text data into multiple rectangles, and each rectangle is sorted in rows and columns according to the order of acquisition to obtain the initial text sequence. For example, a rectangle may include the information {269,322,369,297,446,297,446,322,1,Occupation Other}.

[0094] In step S202 of some embodiments, the line corresponding to each line in the initial text sequence is linearly fitted by the least squares method to obtain the slope value of the line corresponding to each line in the initial text sequence. Then, the slope values ​​of the lines corresponding to all lines are averaged to obtain the line text slope value k, which is the rotation slope of the target image.

[0095] In step S203 of some embodiments, firstly, the center point coordinates (x0, y0) of each rectangle are calculated based on the horizontal and vertical coordinates of the four vertices of the rectangle. Then, based on the slope value of the line text and the center point coordinates, the intercept value b between the matrix box and the y-axis is obtained, where b = y0 - kx0. The intercept value b is calculated for each rectangle. Since the intercept values ​​b of rectangles belonging to the same line segment are almost identical, after calculating the intercept values ​​b of all rectangles, all rectangles are rearranged according to the intercept values ​​b. Specifically, rectangles with intercept value differences less than a preset threshold are grouped into one category. Rectangles of the same category are arranged in ascending order of center point coordinate x0 to obtain the final row of rectangles. The final row of matrix boxes is used as line text data, thereby achieving the row structuring of the original text data.

[0096] Through the above steps S201 to S203, the original structural layout features of the target image can be obtained and preserved when performing text recognition processing on the target image, thereby improving the accuracy of image recognition.

[0097] Before step S104 in some embodiments, the method further includes a pre-trained text classification model, which can be built based on the TextCNN model. The text classification model includes a convolutional network and a combination network. The convolutional network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a pooling layer. The convolutional network is mainly used to extract features from the input vector, obtain the image semantic features of the input vector, and obtain multiple input representation vectors. The combination network is mainly used to integrate and process the multiple input representation vectors, fuse multiple image semantic features, and obtain a fused feature vector. The fused feature vector includes the complete semantic information of the input vector.

[0098] Please see Figure 3 In some embodiments, the text classification model includes convolutional networks and ensemble networks, and step S104 may include, but is not limited to, steps S301 to S304:

[0099] Step S301: Embed the line text data to obtain the line text embedding vector;

[0100] Step S302: Extract features from the line text embedding vector using a convolutional network to obtain the line text representation vector;

[0101] Step S303: Combine the line text representation vectors through a combination network to obtain a fused line text feature vector;

[0102] Step S304: Perform max pooling on the line text representation vector to obtain the line text pooling vector;

[0103] Step S305: The fused line text feature vector and the line text pooling vector are concatenated to obtain the target text representation vector;

[0104] Step S306: Classify the target text representation vector using a preset function to obtain the first classification data.

[0105] In step S301 of some embodiments, each line of text data is embedded to map the line of text data from the semantic space to the vector space to obtain the line text embedding vector.

[0106] In step S302 of some embodiments, the line text embedding vector is processed by two-dimensional convolution through a convolutional network to extract two-dimensional image features of the line text embedding vector, and the extracted two-dimensional image features are processed by max pooling through a pooling layer to obtain the line text representation vector.

[0107] In step S303 of some embodiments, the classification function of the combined network is used to calculate the weights of the line text representation vectors to obtain the classification weight of each line text representation vector. The classification weight can be used to characterize the degree of influence of each line text representation vector on the classification result. The line text representation vectors are weighted according to the classification weight to obtain the fused line text feature vector.

[0108] In step S304 of some embodiments, max pooling is performed on each line of text representation vector, and each line of text representation vector is divided into regions and the maximum value is extracted to obtain the line text pooling vector.

[0109] In step S305 of some embodiments, when concatenating the fused line text feature vector and the line text pooling vector, the fused line text feature vector and the line text pooling vector can be concatenated to obtain the target text representation vector, which can be used to represent the overall semantic content information of the target medical impact.

[0110] In step S306 of some embodiments, the preset function can be a softmax function or a probability function. The softmax function creates a probability distribution for the target text representation vector on each text category label, thereby labeling and classifying the target text representation vector according to the probability distribution, obtaining the classification probability value corresponding to each text category label, and selecting the text category label with the highest classification probability value as the final category label, thus obtaining the first classification data.

[0111] Steps S301 to S304 above can classify line text data based on a text classification model, thereby improving classification accuracy and efficiency.

[0112] Please see Figure 4In some embodiments, the convolutional network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a pooling layer. Step S302 may include, but is not limited to, steps S401 to S404:

[0113] Step S401: The text embedding vector is convolved by the first convolutional layer to obtain the first convolutional vector, and the first convolutional vector is max-pooled by the pooling layer to obtain the first pooling vector.

[0114] Step S402: The line text embedding vector is convolved by the second convolutional layer to obtain the second convolutional vector, and the second convolutional vector is max-pooled by the pooling layer to obtain the second pooling vector.

[0115] Step S403: The line text embedding vector is convolved by three convolutional layers to obtain the third convolutional vector, and the third convolutional vector is max-pooled by a pooling layer to obtain the third pooling vector.

[0116] Step S404: The first pooling vector, the second pooling vector, and the third pooling vector are concatenated to obtain the line text representation vector.

[0117] In step S401 of some embodiments, the line text embedding vector is subjected to two-dimensional convolution processing through the first convolution layer to extract the two-dimensional image features of the line text embedding vector and obtain the first convolution vector. The first convolution vector is then subjected to max pooling processing through the pooling layer to divide the first convolution vector into regions and extract the maximum value to obtain the first pooling vector.

[0118] In step S402 of some embodiments, the line text embedding vector is subjected to two-dimensional convolution processing through the second convolution layer to extract the two-dimensional image features of the line text embedding vector and obtain the second convolution vector. The second convolution vector is then subjected to max pooling processing through the pooling layer to divide the second convolution vector into regions and extract the maximum value to obtain the second pooling vector.

[0119] In step S403 of some embodiments, the line text embedding vector is processed by two-dimensional convolution through the third convolution layer to extract the two-dimensional image features of the line text embedding vector and obtain the third convolution vector. The third convolution vector is then processed by max pooling through the pooling layer to divide the third convolution vector into regions and extract the maximum value to obtain the third pooling vector.

[0120] In step S404 of some embodiments, when concatenating the first pooling vector, the second pooling vector, and the third pooling vector, the first pooling vector, the second pooling vector, and the third pooling vector can be added together or concatenated to obtain a line text representation vector. Each line text representation vector can be used to represent the semantic information of the sentence in the corresponding line of the target image.

[0121] It should be noted that the kernels, sizes, and strides of the first, second, and third convolutional layers can be the same or different, and can be set according to the actual situation without restriction.

[0122] Please see Figure 5 In some embodiments, step S303 may include, but is not limited to, steps S501 to S502:

[0123] Step S501: The classification function of the combined network is used to calculate the weights of the line text representation vectors to obtain the classification weights of each line text representation vector.

[0124] Step S502: The line text representation vector is weighted according to the classification weight to obtain the fused line text feature vector.

[0125] In step S501 of some embodiments, the classification function can be a softmax function, etc. For example, a weight probability distribution is created for each weight value label of the line text representation vector using a softmax function, thereby labeling and classifying the line text representation vector according to the probability distribution to obtain the classification weight of each line text representation vector.

[0126] In step S502 of some embodiments, the text representation vectors of all lines are weighted and summed according to the classification weight of each line text representation vector to obtain the fused line text feature vector.

[0127] Please see Figure 6 In some embodiments, step S306 includes, but is not limited to, steps S601 to S602:

[0128] Step S601: Calculate the classification probability of the target text representation vector using a preset function and preset text category labels to obtain the classification probability value of each text category label;

[0129] Step S602: Obtain the first category data based on the classification probability value.

[0130] In step S601 of some embodiments, the preset function can be a probability function such as a softmax function. The preset text category labels can be set according to the actual business scenario without limitation. For example, text category labels include medicine, disease, symptoms, etc. The softmax function creates a probability distribution for the target text representation vector on each text category label, thereby labeling and classifying the target text representation vector according to the probability distribution to obtain the classification probability value corresponding to each text category label.

[0131] In step S602 of some embodiments, since the magnitude of the classification probability value can intuitively represent the probability that the target text representation vector belongs to each text category label, that is, the larger the classification probability value, the greater the probability that the target text representation vector belongs to that text category label, the text category label with the largest classification probability value is taken as the image category of the target text representation vector to obtain the first classification data. The semantic content of the first classification data is mainly used to represent that the image category is the category to which the target image belongs.

[0132] In step S105 of some embodiments, when classifying the original text data using a preset regular expression matching method, the regular expression used for matching can be set according to the actual scenario without limitation. The regular expression often uses predefined characters and combinations of these specific characters to form a "rule string," where the characters can be letters, numbers, or metacharacters, etc. Regular expressions can be used to express filtering logic larger than a string; that is, by matching the regular expression with one or more strings of the original text data, a matching result is obtained. Based on the matching result, a second category of data is obtained. The semantic content of this second category of data is used to characterize the category to which the original text data (i.e., the target image) belongs. For example, corresponding regular expressions are set for text tags such as medicine, disease, and symptoms. The first regular expression corresponds to the medicine tag, the second regular expression to the disease tag, and the third regular expression to the symptom tag. If a string of some original text data matches the first regular expression, it indicates that the original text data includes content describing medicine and belongs to the medicine tag.

[0133] Furthermore, after multiple experiments, it was verified that when the first five lines of the original text data contain image title text, the classification accuracy using regular expression matching is high, but the recall rate is low. This indicates that, based on the actual text content and layout of the original text data, using regular expression matching can effectively improve the accuracy and reasonableness of image classification.

[0134] Please see Figure 7 In some embodiments, step S106 may include, but is not limited to, steps S701 to S703:

[0135] Step S701: Compare and analyze the first category data and the second category data to obtain the analysis results;

[0136] Step S702: If the analysis result shows that the first category data and the second category data are the same, then the first category data or the second category data shall be used as the target category data.

[0137] Step S703: If the analysis result shows that the first category data and the second category data are different, then obtain the priority of the first category data and the second category data, and use the first category data or the second category data as the target category data according to the priority.

[0138] In step S701 of some embodiments, a comparative analysis is performed on the first category data and the second category data to determine whether the first category data and the second category data are the same, thereby obtaining the analysis result. Specifically, this analysis process mainly compares whether the category labels in the first category data are consistent with the category labels in the second category data.

[0139] In step S702 of some embodiments, if the analysis result shows that the category label in the first classification data is the same as the category label in the second classification data, then the category label in the first classification data or the category label in the second classification data is used as the target category label of the target image, thereby obtaining the target classification data.

[0140] In step S703 of some embodiments, if the analysis result shows that the category label in the first classification data is different from the category label in the second classification data, then the priority of the first classification data and the second classification data is obtained, and the category label in the first classification data or the category label in the second classification data is used as the target category label of the target image according to the priority. For example, if the second classification data is set to high priority and the first classification data is set to low priority, then the classification result of regular expression matching takes precedence over the classification result of the text classification model. When the category label in the first classification data is different from the category label in the second classification data, the category label in the high-priority second classification data is used as the target category label of the target image, thereby obtaining the target classification data.

[0141] Steps S701 to S703 above can combine regular matching and model classification to obtain the category of the target image, thereby effectively improving the accuracy of image classification.

[0142] The image classification method of this application embodiment acquires a target image to be processed; performs text recognition on the target image to obtain original text data, which can conveniently obtain the semantic content information of the target image; further, performs structured processing on the original text data according to a preset algorithm to obtain line text data, which can conveniently obtain the structural layout features of the target image. The line text data is classified using a preset text classification model to obtain first classification data, which improves classification accuracy. Simultaneously, the original text data is classified using a preset regular expression matching method to obtain second classification data, which ensures the rationality of the classification. Finally, based on the first and second classification data, target classification data is obtained. This method combines regular expression matching and model classification to determine the category of the target image, thereby effectively improving the accuracy of image classification.

[0143] Please see Figure 8 This application also provides an image classification apparatus that can implement the above-described image classification method. The apparatus includes:

[0144] Image acquisition module 801 is used to acquire the target image to be processed;

[0145] The text recognition module 802 is used to perform text recognition on the target image to obtain the original text data;

[0146] The structured processing module 803 is used to perform structured processing on the original text data according to a preset algorithm to obtain line text data;

[0147] The first classification module 804 is used to classify the line text data using a preset text classification model to obtain the first classification data;

[0148] The second classification module 805 is used to classify the original text data using a preset regular expression matching method to obtain the second classification data;

[0149] The comparison module 806 is used to obtain the target classification data based on the first classification data and the second classification data.

[0150] In some embodiments, the structured processing module 803 includes:

[0151] The sorting unit is used to sort the original text data using a preset sorting algorithm to obtain an initial text sequence.

[0152] The fitting unit is used to fit the initial text sequence using the least squares method to obtain the line text slope value;

[0153] The line structuring unit is used to perform line structuring processing on the original text data based on the line text slope value to obtain line text data.

[0154] In some embodiments, the text classification model includes a convolutional network and a combination network, and the first classification module 804 includes:

[0155] The embedding unit is used to embed the line text data to obtain the line text embedding vector;

[0156] The extraction unit is used to extract features from the line text embedding vector through a convolutional network to obtain the line text representation vector;

[0157] Combination unit, used to combine line text representation vectors through combination network to obtain fused line text feature vector;

[0158] The pooling unit is used to perform max pooling on the line text representation vector to obtain the line text pooled vector.

[0159] The concatenation unit is used to concatenate the fused line text feature vector and the line text pooling vector to obtain the target text representation vector;

[0160] The classification unit is used to classify the target text representation vector using a preset function to obtain the first classification data.

[0161] In some embodiments, the convolutional network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a pooling layer, and the extraction unit includes:

[0162] The first convolutional subunit is used to perform convolution processing on the line text embedding vector through the first convolutional layer to obtain the first convolutional vector, and to perform max pooling processing on the first convolutional vector through the pooling layer to obtain the first pooled vector.

[0163] The second convolutional subunit is used to perform convolution processing on the line text embedding vector through the second convolutional layer to obtain the second convolutional vector, and then perform max pooling processing on the second convolutional vector through the pooling layer to obtain the second pooled vector.

[0164] The third convolutional subunit is used to perform convolution processing on the line text embedding vector through three convolutional layers to obtain the third convolutional vector, and then perform max pooling processing on the third convolutional vector through a pooling layer to obtain the third pooled vector.

[0165] The concatenation subunit is used to concatenate the first pooling vector, the second pooling vector, and the third pooling vector to obtain the line text representation vector.

[0166] In some embodiments, the combining unit includes:

[0167] The weight calculation subunit is used to calculate the weights of the line text representation vectors by combining the classification functions of the network, and obtain the classification weights of each line text representation vector.

[0168] The weighted calculation subunit is used to perform weighted calculation on the line text representation vector according to the classification weight to obtain the fused line text feature vector.

[0169] In some embodiments, the classification unit includes:

[0170] The probability calculation subunit is used to calculate the classification probability of the target text representation vector using a preset function and preset text category labels, and obtain the classification probability value of each text category label.

[0171] The data sub-determination unit is used to obtain the first classification data based on the classification probability value.

[0172] In some embodiments, the comparison module 806 includes:

[0173] The analysis unit is used to compare and analyze the data in the first category and the data in the second category to obtain the analysis results;

[0174] The first processing unit is used to take either the first category data or the second category data as the target category data if the analysis result is that the first category data and the second category data are the same.

[0175] The second processing unit is used to obtain the priority of the first category data and the second category data if the analysis result is that the first category data and the second category data are different, and to use the first category data or the second category data as the target category data according to the priority.

[0176] The specific implementation of this image classification device is basically the same as the specific implementation of the image classification method described above, and will not be repeated here.

[0177] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned image classification method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0178] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0179] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0180] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the image classification method of the embodiments of this application.

[0181] The input / output interface 903 is used to implement information input and output;

[0182] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0183] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0184] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0185] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the above-described image classification method.

[0186] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0187] The image classification method, image classification device, electronic device, and computer-readable storage medium provided in this application acquire a target image to be processed; perform text recognition on the target image to obtain original text data, which can conveniently obtain the semantic content information of the target image; further, perform structured processing on the original text data according to a preset algorithm to obtain line text data, which can conveniently obtain the structural layout features of the target image. Classify the line text data using a preset text classification model to obtain first classification data, which can improve classification accuracy based on the classification model; simultaneously, classify the original text data using a preset regular expression matching method to obtain second classification data, which can improve the rationality of classification; finally, obtain target classification data based on the first and second classification data, which can combine the two cases of regular expression matching and model classification to obtain the category to which the target image belongs, thereby effectively improving the accuracy of image classification.

[0188] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0189] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0190] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0191] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0192] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0193] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0195] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0197] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An image classification method, characterized in that, The method includes: Obtain the target image to be processed; Perform text recognition on the target image to obtain the original text data; The original text data is structured according to a preset algorithm to obtain line text data; The line text data is classified using a preset text classification model to obtain the first category data; The original text data is classified using a preset regular expression matching method to obtain second-classified data; Based on the first classification data and the second classification data, the target classification data is obtained; The text classification model includes convolutional networks and ensemble networks. The process of classifying the line text data using the preset text classification model to obtain first-classified data includes: The line text data is embedded to obtain a line text embedding vector; features are extracted from the line text embedding vector using the convolutional network to obtain a line text representation vector; the line text representation vector is combined using the combining network to obtain a fused line text feature vector; the line text representation vector is max-pooled to obtain a line text pooling vector; the fused line text feature vector and the line text pooling vector are concatenated to obtain a target text representation vector; the target text representation vector is classified using a preset function to obtain the first classification data. The step of combining the line text representation vectors through the combined network to obtain the fused line text feature vector includes: The classification function of the combined network is used to calculate the weights of the line text representation vectors to obtain the classification weight of each line text representation vector; the line text representation vectors are then weighted according to the classification weights to obtain the fused line text feature vector.

2. The image classification method according to claim 1, characterized in that, The step of performing structured processing on the original text data according to a preset algorithm to obtain line text data includes: The original text data is sorted using a preset sorting algorithm to obtain an initial text sequence; The initial text sequence is fitted using the least squares method to obtain the line text slope value; The original text data is processed by line structuring based on the line text slope value to obtain the line text data.

3. The image classification method according to claim 1, characterized in that, The convolutional network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a pooling layer. The step of extracting features from the line text embedding vector through the convolutional network to obtain the line text representation vector includes: The first convolutional layer performs convolution processing on the line text embedding vector to obtain a first convolutional vector, and the pooling layer performs max pooling processing on the first convolutional vector to obtain a first pooling vector. The second convolutional layer performs convolution processing on the line text embedding vector to obtain a second convolutional vector, and the pooling layer performs max pooling processing on the second convolutional vector to obtain a second pooling vector. The line text embedding vector is convolved by the three convolutional layers to obtain the third convolutional vector, and then the third convolutional vector is max-pooled by the pooling layer to obtain the third pooling vector. The first pooling vector, the second pooling vector, and the third pooling vector are concatenated to obtain the line text representation vector.

4. The image classification method according to claim 1, characterized in that, The step of classifying the target text representation vector using a preset function to obtain the first classification data includes: The classification probability of the target text representation vector is calculated by using the preset function and preset text category labels to obtain the classification probability value of each text category label; The first classification data is obtained based on the classification probability value.

5. The image classification method according to any one of claims 1 to 4, characterized in that, The step of obtaining the target classification data based on the first classification data and the second classification data includes: The first category data and the second category data are compared and analyzed to obtain the analysis results; If the analysis result shows that the first category data and the second category data are the same, then the first category data or the second category data shall be used as the target category data. If the analysis result shows that the first category data and the second category data are different, then the priority of the first category data and the second category data is obtained, and the first category data or the second category data is used as the target category data according to the priority.

6. An image classification device, characterized in that, The device includes: The image acquisition module is used to acquire the target image to be processed; The text recognition module is used to perform text recognition on the target image to obtain the original text data; The structured processing module is used to perform structured processing on the original text data according to a preset algorithm to obtain line text data; The first classification module is used to classify the line text data using a preset text classification model to obtain the first classification data; The second classification module is used to classify the original text data using a preset regular expression matching method to obtain second classification data; The comparison module is used to obtain target classification data based on the first classification data and the second classification data; The text classification model includes convolutional networks and ensemble networks. The process of classifying the line text data using the preset text classification model to obtain first-classified data includes: The line text data is embedded to obtain a line text embedding vector; features are extracted from the line text embedding vector using the convolutional network to obtain a line text representation vector; the line text representation vector is combined using the combining network to obtain a fused line text feature vector; the line text representation vector is max-pooled to obtain a line text pooling vector; the fused line text feature vector and the line text pooling vector are concatenated to obtain a target text representation vector; the target text representation vector is classified using a preset function to obtain the first classification data. The step of combining the line text representation vectors through the combined network to obtain the fused line text feature vector includes: The classification function of the combined network is used to calculate the weights of the line text representation vectors to obtain the classification weight of each line text representation vector; the line text representation vectors are then weighted according to the classification weights to obtain the fused line text feature vector.

7. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the image classification method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the image classification method according to any one of claims 1 to 5.

Images