Image recognition method, device, equipment and storage medium

Abstract

The application discloses an image recognition method, device and equipment and a storage medium, and belongs to the technical field of computers. Through the technical scheme provided by the embodiment of the application, when image recognition is performed, a plurality of image blocks of an image to be recognized are obtained, feature extraction is performed on the plurality of image blocks respectively, and a plurality of image block features are obtained. When the image feature of the image is obtained, the plurality of image block features are fused, compared with a direct extraction mode of the image feature of the image, the image feature obtained by adopting the technical scheme provided by the embodiment of the application has stronger expression capability, and when the category of the image is determined based on the image feature, more accurate results can be obtained.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image recognition method, apparatus, device, and storage medium. Background Technology

[0002] With the development of computer technology, artificial intelligence technology is booming. As a branch of artificial intelligence, image recognition technology is being applied in a wider range of applications. For example, image recognition technology can be applied to facial recognition scenarios, where image recognition models are used to identify images containing faces to obtain identity information corresponding to the faces; or it can be applied to medical scenarios, where image recognition models are used to identify medical images and discover some lesions that are not easily distinguishable by the naked eye, thereby assisting doctors in determining treatment plans.

[0003] In related technologies, image recognition models often directly extract features from the image to be recognized, obtain the feature map of the image, and determine the category of the image based on the feature map.

[0004] However, different regions within an image may contribute differently to determining the image category. For example, in medical image recognition, lesions may be small and distributed in different locations. Therefore, the accuracy of image recognition using methods from related technologies is not high. Summary of the Invention

[0005] This application provides an image recognition method, apparatus, device, and storage medium, which can improve the accuracy of image recognition. The technical solution is as follows:

[0006] On the one hand, an image recognition method is provided, the method comprising:

[0007] Obtain multiple image patches of the image to be identified;

[0008] Feature extraction is performed on the plurality of image patches to obtain a plurality of image patch features, wherein one image patch feature corresponds to one image patch;

[0009] The multiple image patch features are subjected to dimensionality reduction processing to obtain multiple dimensionality-reduced features;

[0010] The multiple dimensionality reduction features are fused to obtain the image features of the image;

[0011] Based on the image features, the category of the image is determined.

[0012] On the one hand, an image recognition method is provided, the method comprising:

[0013] Obtain multiple sample image patches from the sample image;

[0014] Feature extraction is performed on the plurality of sample image blocks to obtain a plurality of sample image block features, wherein one sample image block feature corresponds to one sample image block;

[0015] The multiple sample image patch features are input into an image recognition model. The image recognition model performs dimensionality reduction on the multiple sample image patch features to obtain multiple dimensionality-reduced features. The multiple dimensionality-reduced features are fused to obtain the sample image features of the sample image. Based on the sample image features, the first predicted category of the sample image is output.

[0016] The image recognition model is trained based on the first difference information between the first predicted category of the sample image and the actual category of the sample image.

[0017] In one possible implementation, obtaining multiple attention weights based on the multiple dimensionality reduction features includes:

[0018] The multiple dimensionality-reduced features are subjected to fully connected processing, random deactivation processing, standardization processing, and normalization processing to obtain the multiple attention weights.

[0019] In one possible implementation, before training the image recognition model based on the first difference information between the first predicted category of the sample image and the actual category of the sample image, the method further includes:

[0020] The features of the multiple sample image blocks are subjected to fully connected processing and normalization processing to obtain the second predicted category of the sample image;

[0021] Training the image recognition model based on the first difference information between the first predicted category and the actual category of the sample image includes:

[0022] The image recognition model is trained based on the first difference information and the second difference information, wherein the second difference information is the difference information between the second predicted category and the actual category.

[0023] On one hand, an image recognition device is provided, the device comprising:

[0024] The image patch acquisition module is used to acquire multiple image patches of the image to be identified;

[0025] The first feature extraction module is used to extract features from the plurality of image blocks to obtain a plurality of image block features, wherein one image block feature corresponds to one image block;

[0026] The dimensionality reduction module is used to perform dimensionality reduction processing on the multiple image patch features to obtain multiple dimensionality-reduced features;

[0027] The feature fusion module is used to fuse the multiple dimensionality-reduced features to obtain the image features of the image;

[0028] A category determination module is used to determine the category of the image based on the image features.

[0029] In one possible implementation, the first feature extraction module is used to encode the plurality of image patches respectively based on an attention mechanism to obtain the features of the plurality of image patches.

[0030] In one possible implementation, the dimensionality reduction module is used to map the features of the plurality of image blocks to a feature space through an image recognition model to obtain the plurality of dimensionality-reduced features in the feature space, wherein the dimension of the feature space is lower than that of the space in which the plurality of image block features are located.

[0031] In one possible implementation, the dimensionality reduction module is used to perform linear transformations on the multiple image patch features respectively to obtain multiple target linear transformation features in the feature space, where each target linear transformation feature corresponds to a dimensionality reduction feature; and to perform standardization and activation processing on the multiple target linear transformation features to obtain the multiple dimensionality reduction features.

[0032] In one possible implementation, the feature fusion module is used to obtain multiple attention weights based on the multiple dimensionality reduction features, wherein one attention weight corresponds to one of the dimensionality reduction features;

[0033] Based on the multiple attention weights, the corresponding dimensionality reduction features are weighted and summed to obtain the image features of the image.

[0034] In one possible implementation, the feature fusion module is used to perform fully connected processing, standardization processing, and normalization processing on the plurality of dimensionality-reduced features to obtain the plurality of attention weights.

[0035] In one possible implementation, the category determination module is used to classify the image features to obtain a probability distribution column of the image, wherein the values ​​in the probability distribution column are used to represent the probability that the image corresponds to a category;

[0036] In response to any value in the probability distribution column representing a probability that meets the target probability condition, the category corresponding to the given value is determined as the category of the image.

[0037] On one hand, an image recognition device is provided, the device comprising:

[0038] The sample image patch acquisition module is used to acquire multiple sample image patches from a sample image.

[0039] The second feature extraction module is used to extract features from the plurality of sample image blocks to obtain a plurality of sample image block features, wherein one sample image block feature corresponds to one sample image block.

[0040] The input module is used to input the features of the multiple sample image blocks into the image recognition model, and through the image recognition model, perform dimensionality reduction processing on the features of the multiple sample image blocks to obtain multiple dimensionality-reduced features; fuse the multiple dimensionality-reduced features to obtain the sample image features of the sample image; and output the first predicted category of the sample image based on the sample image features.

[0041] The training module is used to train the image recognition model based on the first difference information between the first predicted category of the sample image and the actual category of the sample image.

[0042] In one possible implementation, the input module is used to map the features of the plurality of sample image blocks to a feature space to obtain the plurality of dimensionality-reduced features in the feature space, wherein the dimension of the feature space is lower than that of the space in which the features of the plurality of sample image blocks reside.

[0043] In one possible implementation, the input module is used to perform linear transformations on the plurality of dimensionality-reduced features respectively to obtain a plurality of target linear transformation features in the feature space, wherein one target linear transformation feature corresponds to one dimensionality-reduced feature;

[0044] The multiple target linear transformation features are standardized to obtain the multiple dimensionality-reduced features.

[0045] In one possible implementation, the input module is configured to obtain multiple attention weights based on the multiple dimensionality reduction features, wherein one attention weight corresponds to one dimensionality reduction feature; and to perform a weighted summation of the corresponding dimensionality reduction features based on the multiple attention weights to obtain the sample image features of the sample image.

[0046] In one possible implementation, the input module is used to perform fully connected processing, random deactivation processing, standardization processing, and normalization processing on the plurality of dimensionality-reduced features to obtain the plurality of attention weights.

[0047] In one possible implementation, the device further includes:

[0048] The prediction category determination module is used to perform fully connected processing and normalization processing on the features of the multiple sample image blocks to obtain the second predicted category of the sample image;

[0049] The training module is used to train the image recognition model based on the first difference information and the second difference information, wherein the second difference information is the difference information between the second predicted category and the actual category.

[0050] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the image recognition method.

[0051] On one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer program, which is loaded and executed by a processor to implement the image recognition method.

[0052] On one hand, a computer program product or computer program is provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the image recognition method described above.

[0053] The technical solution provided in this application, when performing image recognition, first acquires multiple image blocks of the image to be recognized, and then extracts features from each of the multiple image blocks to obtain multiple image block features. When acquiring image features, a method of fusing multiple image block features is adopted. Compared to directly extracting image features, the image features obtained using the technical solution provided in this application have stronger expressive power, and thus, more accurate results can be obtained when determining the image category based on the image features. Attached Figure Description

[0054] To illustrate the technical solutions in the embodiments of this application more clearly, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of the implementation environment of an image recognition method provided in an embodiment of this application;

[0056] Figure 2 This is a schematic diagram of the structure of an image recognition model provided in an embodiment of this application;

[0057] Figure 3 This is a flowchart of an image recognition method provided in an embodiment of this application;

[0058] Figure 4 This is a flowchart of an image recognition method provided in an embodiment of this application;

[0059] Figure 5 This is a schematic diagram of an interface provided in an embodiment of this application;

[0060] Figure 6 This is a flowchart of an image recognition method provided in an embodiment of this application;

[0061] Figure 7 This is a flowchart of an image recognition method provided in an embodiment of this application;

[0062] Figure 8 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application;

[0063] Figure 9 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application;

[0064] Figure 10 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0065] Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0067] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0068] In this application, the term "at least one" means one or more, and "multiple" means two or more, for example, multiple reference face images means two or more reference face images.

[0069] Artificial intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0070] Computer vision (CV) is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and then performing image processing to make the computer-processed images more suitable for human observation or transmission to instruments for detection.

[0071] The technical solutions provided in this application can also be combined with cloud technology, for example, by deploying the trained image recognition model on a cloud server. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0072] Among them, medical cloud in cloud technology refers to the creation of a healthcare service cloud platform using cloud computing, mobile technology, multimedia, 4G communication, big data, and the Internet of Things, combined with medical technology, thereby enabling the sharing of medical resources and expanding the scope of medical care. Because of the application and integration of cloud computing technology, medical cloud improves the efficiency of medical institutions and facilitates access to medical care for residents. For example, hospital appointment booking and medical insurance are products of the integration of cloud computing and the medical field. Medical cloud also has advantages such as data security, information sharing, dynamic expansion, and global deployment. For instance, the image recognition model provided in this application embodiment is deployed on a healthcare service cloud platform.

[0073] Multiple-instance learning (MIL) is a method evolved from supervised learning algorithms. It defines a "bag" as a collection of multiple instances and has wide applications. Instead of receiving a set of individually labeled instances, the learner receives a set of labeled bags, each containing multiple instances. In the simple case of MIL binary classification, a bag is labeled negative if all instances in it are negative. Conversely, a bag is labeled positive if at least one instance in it is positive.

[0074] Transformer: A neural network based on its own attention mechanism, widely used in speech recognition, image recognition, natural language processing, and other fields. It generates a new output sequence by adding weights to the input sequence through an attention mechanism one or more times, weakening or forgetting content that does not conform to the attention model.

[0075] Normalization: Mapping sequences with different value ranges to the interval (0, 1) to facilitate data processing. In some cases, normalized values ​​can be directly expressed as probabilities.

[0076] Dropout is a method for optimizing deep artificial neural networks. During the learning process, it reduces the interdependence between nodes by randomly setting some weights or outputs of the hidden layers to zero, thereby standardizing the neural network and reducing its structural risks. For example, in model training, given a vector (1, 2, 3, 4), after inputting this vector into a dropout layer, the dropout layer can randomly convert one of the numbers in the vector (1, 2, 3, 4) to 0. For example, converting 2 to 0 would change the vector to (1, 0, 3, 4).

[0077] Learning rate: Used to control the learning progress of the model. The learning rate guides the model in gradient descent on how to adjust network weights using the gradient of the loss function. If the learning rate is too large, the loss function may directly skip the global optimum, resulting in excessive loss. If the learning rate is too small, the loss function changes very slowly, greatly increasing the convergence complexity of the network and making it easy to get trapped in local minima or saddle points.

[0078] Embedded coding, mathematically speaking, represents a correspondence, that is, mapping data in space X to space Y using a function F. This function F is injective, and the mapping result preserves the structure. An injective function means that the mapped data uniquely corresponds to the original data, and preserving the structure means that the order of the original data remains the same after mapping. For example, if there are data X1 and X2 before mapping, after mapping we get Y1 corresponding to X1 and Y2 corresponding to X2. If the original data X1 > X2, then correspondingly, the mapped data Y1 > Y2. For words, this means mapping words to another space to facilitate subsequent machine learning and processing.

[0079] Attention weights represent the importance of a piece of data during training or prediction. Importance indicates the magnitude of the influence of input data on output data. Data with high importance corresponds to higher attention weights, while data with low importance corresponds to lower attention weights. The importance of data varies in different scenarios, and training the model to assign attention weights is essentially the process of determining data importance.

[0080] Optionally, the computer device provided in this application embodiment can be implemented as a terminal or a server. The implementation environment consisting of a terminal and a server will be described below.

[0081] Figure 1 This is a schematic diagram illustrating the implementation environment of an image recognition method provided in this application embodiment. See also... Figure 1 The implementation environment may include terminal 110 and server 140.

[0082] Terminal 110 is connected to server 140 via a wireless or wired network. Optionally, terminal 110 may be a smartphone, tablet, laptop, desktop computer, smartwatch, etc., but is not limited to these. Terminal 110 has an application that supports image recognition installed and running.

[0083] Optionally, server 140 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. In some embodiments, a user can upload an image to be recognized to server 140 via terminal 110, and server 140 will execute the image recognition method provided in this application embodiment on the uploaded image. After recognition is completed, server 140 will send the recognition result to terminal 110, and the user can view the image recognition result through terminal 110.

[0084] Optionally, terminal 110 refers to one of a plurality of terminals. This application embodiment only uses terminal 110 as an example for illustration.

[0085] Those skilled in the art will understand that the number of terminals 110 described above can be more or less. For example, there may be only one terminal 110, or there may be dozens or hundreds of terminals 110, or even more. In this case, other terminals are also included in the above implementation environment. The embodiments of this application do not limit the number of terminals or the type of devices.

[0086] After introducing the implementation environment of the image recognition method provided in the embodiments of this application, the application scenarios of the image recognition method provided in the embodiments of this application will be described below. It should be noted that the computer device in the following description process can be implemented as the terminal 110 or the server 140 in the above implementation environment, and this application embodiment does not limit it in this way.

[0087] The image recognition method provided in this application can be applied to various image recognition scenarios. For example, it can be applied to the recognition of images of eye diseases, the recognition of images of buildings, the recognition of images of vehicles, or the recognition of images of materials, etc. This application does not limit the application to these scenarios.

[0088] In scenarios involving the identification of eye disease images, such as determining the presence of diabetic retinopathy in the eye disease images of diabetic patients, determining the type of fundus disease corresponding to the patient's eye disease image, or determining the type of glaucoma corresponding to the eye disease image of a glaucoma patient, this application embodiment does not limit the scope of application. Since identifying the disease category corresponding to an eye disease image requires considerable experience, and disease category identification is the foundation of effective treatment, less experienced medical personnel can utilize the image recognition method provided in this application embodiment to identify the disease category corresponding to the eye disease image, thereby improving the accuracy of disease category identification. When eye disease image identification is required, medical personnel upload the eye disease image to the server via a terminal. The server then executes the image recognition method provided in this application embodiment on the received eye disease image to determine the type of eye disease corresponding to the image. The server returns the eye disease corresponding to the image to the terminal, which then displays the identification result to the medical personnel. This identification result can serve as a reference for medical personnel in disease identification, thereby assisting them in formulating treatment plans.

[0089] In scenarios involving material image recognition, such as identifying cracks in metallic materials, when a crack exists, the impact of external force is concentrated at the crack's location, leading to crack propagation and material failure. When this metallic material serves as a structural support, this aging process can have severe consequences. However, the initial crack size in metallic materials is often small, making it difficult for the human eye to effectively detect. In such cases, users can upload images of the metallic material to a server via their terminals. The server then executes the image recognition method provided in this embodiment to determine the presence of cracks in the image. The server returns the recognition result to the terminal, which then displays the result to the user, allowing them to promptly identify cracks that are difficult for the human eye to detect.

[0090] In scenarios involving building image recognition, such as identifying whether a certain type of building exists within a set of images, the image recognition method provided in this application can be employed. For example, consider a set of building images containing four types of buildings: Type A, Type B, Type C, and Type D. Each image may contain only one type of building or multiple types. If the goal of image recognition is to identify images containing Type C buildings from this set of images, manual identification would be inefficient. However, the technical solution provided in this application allows for rapid identification of images containing Type C buildings from this set of images. Users can upload the set of building images to a server via a terminal. The server then executes the image recognition method provided in this application on the set of building images to determine whether an image containing a Type C building exists within the set. The server returns the recognition result to the terminal, which then displays the result to the user, allowing the user to identify images containing Type C buildings from the set of building images.

[0091] In scenarios involving vehicle image recognition, such as identifying whether a certain type of vehicle exists within a set of images, the image recognition method provided in this application can be employed. For example, consider a set of vehicle images containing four vehicle types: E-type, F-type, G-type, and H-type. Each image may contain only one type of vehicle or multiple types. If the goal of image recognition is to identify images containing G-type vehicles from this set of images, manual identification would be inefficient. However, the technical solution provided in this application allows for rapid identification of images containing G-type vehicles from this set of images. Users can upload the set of vehicle images to a server via their terminal. The server then executes the image recognition method provided in this application on the set of vehicle images to determine whether an image containing a G-type vehicle exists. The server returns the recognition result to the terminal, which then displays the result to the user, allowing the user to identify images containing G-type vehicles from the set of vehicle images.

[0092] In this embodiment of the application, a computer device can implement the image recognition method provided in this embodiment of the application by means of an image recognition model. The following will be combined with... Figure 2 The structure of the image recognition model provided in the embodiments of this application will be described.

[0093] See Figure 2 The image recognition model 200 includes a feature dimensionality reduction layer 201, an image feature acquisition layer 202, a feature fusion layer 203, and an image recognition layer 204.

[0094] The feature reduction layer 201 is used to obtain the dimensionality reduction features corresponding to the image patch features of multiple image patches. The image feature acquisition layer 202 is used to obtain the image features of the image based on the multiple dimensionality reduction features. The image recognition layer 204 is used to perform recognition based on the image features of the image to obtain the category of the image.

[0095] In some embodiments, the feature dimensionality reduction layer 201 includes a first linear layer 2011, a first layer normalization 2012, and a first activation layer 2013. The first linear layer 2011 performs a linear transformation on the image patch features to achieve dimensionality reduction. The first normalization layer 2012 and the first activation layer 2013 perform normalization and activation processing on the dimensionality-reduced image patch features, respectively, to obtain the dimensionality-reduced features corresponding to each image patch. The normalization process transforms the multiple values ​​corresponding to each feature into values ​​that follow a normal distribution with a mean of 0 and a variance of 1.

[0096] Activation processing is also known as nonlinear transformation. For example, nonlinear transformations can be performed using ReLU (Rectified Linear Unit) function, Sigmoid (S-shaped growth curve) function, Tanh (tangent) function, etc., to improve the expressive power of image recognition models.

[0097] In some embodiments, the image feature acquisition layer 202 includes a second linear layer 2021, a second normalization layer 2022, a second activation layer 2023, a random deactivation layer 2024, a third linear layer 2025, and a normalization layer 2026. The second linear layer 2021 and the third linear layer 2025 are used to perform linear transformations on the features. The second normalization layer 2022 is used to normalize the features. The random deactivation layer 2024 is used to randomly deactivate the features during training of the image recognition model. Random deactivation involves randomly setting the value of a feature to 0, which avoids overfitting in the image recognition model and improves its generalization ability. The normalization layer 2026 is used to normalize the features to obtain the attention weights corresponding to the image patches.

[0098] The feature fusion layer 203 is used to fuse the dimensionality-reduced features of the image patches based on the attention weights corresponding to the image patches to obtain the image features.

[0099] In some embodiments, the image recognition layer 204 is a classifier. After inputting the image features of the image into the image recognition layer 204, the probability distribution column of the image can be obtained. Each number in the distribution column is used to represent the probability that the image corresponds to a category.

[0100] It should be noted that, Figure 2 The structure of the image recognition model shown is merely an example. In other possible implementations, the image recognition model may have other structures, and this application does not limit this.

[0101] After introducing the implementation environment, application scenarios, and structure of the image recognition model in the embodiments of this application, the image recognition method provided in the embodiments of this application will be described below. In the embodiments of this application, the image recognition method provided in the embodiments of this application can be implemented by a server or a terminal as the execution subject, or it can be implemented through the interaction between the terminal and the server. The terminal is the terminal 110 in the above-described implementation environment, and the server is the server 140 in the above-described implementation environment. The interaction between the terminal and the server means that the terminal sends a sample dataset to the server, the server trains the image recognition model, and the server returns the trained image recognition model to the terminal. The embodiments of this application do not limit the execution subject. The following description will use a server as the execution subject as an example. See [link to documentation]. Figure 3 The methods include:

[0102] Figure 3 This is a flowchart of an image recognition method provided in an embodiment of this application. See also... Figure 3 The methods include:

[0103] 301. The server obtains multiple image blocks of the image to be identified.

[0104] The number of image blocks is related to the size of the image to be identified and the size of the image blocks. If the size of the image to be identified is 16×16 and the size of the image blocks is 8×8, then the server can obtain 4 image blocks of the image to be identified. The image to be identified can be any one of the following: an image of an eye disease, an image of a building, an image of a vehicle, or an image of a material. This application embodiment does not limit this.

[0105] In some embodiments, the image to be identified in this application is stored on a blockchain.

[0106] 302. The server extracts features from multiple image patches to obtain multiple image patch features, with one image patch feature corresponding to one image patch.

[0107] The process of extracting features from an image patch is also the process of abstracting a feature vector from the image patch, and the feature vector can represent the features of the image patch.

[0108] 303. The server performs dimensionality reduction processing on multiple image patch features to obtain multiple dimensionality-reduced features.

[0109] Dimensionality reduction is a method to reduce the dimension of image patch features. For example, if the server uses vectors to represent image patch features, and the size of the vector corresponding to an image patch feature is 1×D, then by performing dimensionality reduction on the image patch feature, we can obtain the vector corresponding to the dimensionality reduction feature of the image patch. The size of the vector corresponding to the dimensionality reduction feature is 1×M, where D and M are both positive integers and M < D.

[0110] 304. The server fuses multiple dimensionality reduction features to obtain the image features of the image.

[0111] Since different dimensionality reduction features can represent the image features of different regions of the image, fusing multiple dimensionality reduction features can yield the complete features of the image.

[0112] 305. The server determines the category of an image based on its image features.

[0113] In this context, for images of eye diseases, the category refers to the disease category corresponding to the image. For images of buildings, the category refers to the building category corresponding to the image. For images of vehicles, the category refers to the vehicle category corresponding to the image; however, this embodiment does not impose any limitations on these categories.

[0114] The technical solution provided in this application provides that, during image recognition, multiple image blocks of the image to be recognized are acquired, and feature extraction is performed on each of the multiple image blocks to obtain multiple image block features. When acquiring image features, a method of fusing multiple image block features is adopted. Compared to directly extracting image features, the image features obtained using the technical solution provided in this application have stronger expressive power, and thus, more accurate results can be obtained when determining the image category based on the image features.

[0115] It should be noted that steps 301-305 above are a brief introduction to the image recognition method provided in the embodiments of this application. The image recognition method provided in the embodiments of this application will be described in more detail below with some examples. See also Figure 4 The methods include:

[0116] Figure 4 This is a flowchart of an image recognition method provided in an embodiment of this application. See also... Figure 4 The methods include:

[0117] 401. The server obtains the image to be identified.

[0118] In one possible implementation, the server acquires the image to be identified uploaded by the terminal. Here, the server is server 140 in the aforementioned implementation environment, and the terminal is terminal 110 in the same environment.

[0119] In this implementation, users can upload the images they want to recognize to the server via their terminals, and the server will then recognize the images. In this case, the server also provides a cloud service for image recognition, and terminals that establish a network connection with the server can use this cloud service.

[0120] For example, a terminal acquires an image to be recognized using a camera and sends an image recognition request to a server, the request carrying the image. The server receives the image recognition request and retrieves the image from it.

[0121] The above implementation methods will be described below in different scenarios.

[0122] Scenario 1: In a scenario involving image recognition of eye diseases, the imaging device is an Optical Coherence Tomography (OCT) scanner. The user (healthcare worker) can control the OCT scanner via a terminal to capture multiple images of the patient's eyes, obtaining multiple images of the patient's eye disease. The terminal also serves as the control terminal for the OCT scanner. After acquiring multiple images, the user can select one to be recognized. The selected image is then sent to the server via the terminal, carrying the image of the eye disease. The server receives the image recognition request and retrieves the image of the eye disease from it.

[0123] For example, see Figure 5 The terminal runs an application for image recognition, which provides an image recognition interface 501. The image recognition interface 501 displays an image selection control 502. In response to a click on the image selection control 502, the terminal displays an image selection area 503 on the image recognition interface, which displays multiple eye disease images captured by an OCT device. In response to selecting any eye disease image, the terminal sends an image recognition request to the server, the request carrying the selected eye disease image. The server receives the image recognition request and retrieves the eye disease image from it.

[0124] Scenario 2: In a scenario involving building image recognition, the capturing device can be any one of a camera, a drone's camera, or a terminal's camera. Taking a smartphone as the terminal and its camera as the capturing device, a user can capture multiple building images using the smartphone's camera. After acquiring these images, the user can select one; the selected image is the one to be recognized. The user can send an image recognition request, containing the building image, to the server via their smartphone. The server receives the image recognition request and retrieves the building image from it.

[0125] It should be noted that in scenarios involving building image recognition, the terminal can also run an image recognition application as described in Scenario 1. Users can use this application to select the image to be recognized.

[0126] Scenario 3: In a scenario involving material image recognition, the imaging device is any one of a Scanning Electron Microscope (SEM), an Optical Microscope (OM), or a camera. Taking an OM as the imaging device and a metallic material as an example, for a batch of metallic materials, the user can control the OM via a terminal to capture images of the batch of metallic materials, obtaining multiple images of the metallic materials. The user can select from these multiple images; the selected image is the one to be recognized. The user can send an image recognition request to the server via the terminal, which carries the image of the metallic material. The terminal is the one connected to the OM. The server receives the image recognition request and retrieves the image of the metallic material from it.

[0127] It should be noted that in scenarios involving material image recognition, the terminal can also run an image recognition application as described in Scenario 1. Users can use this application to select the image to be recognized.

[0128] In one possible implementation, the server loads the image to be identified from a corresponding maintained database.

[0129] For example, a server maintains a database storing multiple images uploaded by users, and the server loads the image to be recognized from these images. For instance, a terminal may run an image recognition application, through which the user can upload images to the server's database. In some embodiments, the application can display multiple images stored in the database. When image recognition is needed for a particular uploaded image, the user can select an image from the database using the application; the selected image is the one to be recognized. In response to the selection of any image stored in the database, the terminal sends an image retrieval request to the server through the application, carrying an identifier for the selected image. Upon receiving the image retrieval request, the server retrieves the identifier from the request, queries the database based on the identifier, and retrieves the image corresponding to that identifier; this image is the one to be recognized.

[0130] 402. The server obtains multiple image blocks of the image to be identified.

[0131] In one possible implementation, the server performs image segmentation on the image to be identified, obtaining multiple image blocks of the image.

[0132] For example, the server divides the pixel value matrix of the image to be identified into multiple sub-pixel value matrices based on the size of the image and the size of the image blocks, with each sub-pixel value matrix corresponding to an image block. The size of the image blocks is set by technicians according to actual conditions, and this embodiment does not limit this.

[0133] For example, if the pixel value matrix of an image to be recognized is That is, the size of the image to be identified is 4×4. If the size of the image patch is 2×2, then the server will process the pixel value matrix of the image. The data is segmented to obtain four sub-pixel value matrices. , , as well as These four sub-pixel value matrices are the sub-pixel value matrices corresponding to the four image blocks of the image.

[0134] Based on the above example, optionally, in order to simplify the calculation, after obtaining the sub-pixel value matrices corresponding to multiple image blocks respectively, the server performs a shape reshape operation on the sub-pixel value matrix corresponding to each image block to obtain a one-dimensional vector corresponding to each image block.

[0135] For example, for sub-pixel value matrix In other words, the server will store the sub-pixel value matrix Directly "flattening" means applying the sub-pixel value matrix. After reshaping the shape, a one-dimensional vector (75, 90, 35, 93) is obtained. This one-dimensional vector (75, 90, 35, 93) can represent the sub-pixel value matrix. The server can perform subsequent operations based on this one-dimensional vector. Compared with high-dimensional matrices, one-dimensional vectors can reduce the amount of computation on the server and improve computational efficiency.

[0136] In some embodiments, the size of the image to be identified is not an integer multiple of the size of the image patch. For example, the size of the image to be identified is 5×5, and the size of the image patch is 2×2. In this case, the server can pad the pixel value matrix of the image, that is, fill the area around the pixel value matrix with 0s or 1s, so that the size of the padded pixel value matrix is ​​an integer multiple of the size of the image patch. The server then divides the padded pixel value matrix into multiple sub-pixel matrices, that is, divides the image into multiple image patches.

[0137] For example, if the pixel value matrix of an image to be recognized is That is, if the size of the image to be identified is 5×5, and the size of the image patch is 2×2, then the server can process the pixel value matrix as follows: Perform a fill operation, taking filling the pixel value matrix with 0s as an example, to obtain the filled pixel value matrix. The server, based on the size of the padded pixel value matrix (6×6) and the image patch size (2×2), performs calculations on the padded pixel value matrix. The pixels are divided into 9 sub-pixel matrices. , , , , , , , as well as .

[0138] Of course, after the server obtains multiple sub-pixel value matrices based on the filled pixel value matrix, it can also perform shape reshaping (reshape operation) on each of the multiple sub-pixel value matrices to obtain a one-dimensional vector corresponding to each image block. The implementation process is described in the previous description and will not be repeated here.

[0139] 403. The server extracts features from multiple image patches to obtain multiple image patch features, with one image patch feature corresponding to one image patch.

[0140] In one possible implementation, the server encodes multiple image patches separately based on an attention mechanism to obtain multiple image patch features.

[0141] For example, the server performs embedding encoding on multiple image patches, obtaining multiple embedding vectors. Each embedding vector corresponds to one image patch, and the embedding vector of an image patch represents the content and position of the image patch within the image. The server inputs these multiple embedding vectors into a feature extraction model. The model uses three linear transformation matrices to perform linear transformations on the embedding vectors, obtaining a query vector, a key vector, and a value vector for each image patch. Based on the query and key vectors corresponding to the multiple image patches, the server uses the feature extraction model to obtain attention weights for each image patch. Finally, based on the attention weights and value vectors of each image patch, the server uses the feature extraction model to obtain an attention encoding vector for each image patch; this attention encoding vector is the image patch feature. In this implementation, the feature extraction model is a Transformer encoder.

[0142] To illustrate the above examples more clearly, the following explanation will be divided into three parts.

[0143] The first part explains the method by which the server performs linear transformations on multiple embedding vectors using three linear transformation matrices of the feature extraction model to obtain the query vector, key vector, and value vector corresponding to each image patch.

[0144] In one possible implementation, the server uses a feature extraction model to multiply each embedding vector by three linear transformation matrices to obtain the query vector, key vector, and value vector corresponding to each image patch.

[0145] For example, for one image patch (1, 3, 4) among multiple image patches, the server uses the three linear transformation matrices WQ of the feature extraction model. WK and WV Multiply the image patch (1, 3, 4) on the right to obtain the query vector (2, 3, 15), key vector (5, 10, 1), and value vector (-8, 18, 6) for the image patch (1, 3, 4).

[0146] Part Two explains the method by which the server obtains the attention weights of multiple image patches based on the query vectors and key vectors corresponding to multiple image patches through a feature extraction model.

[0147] In one possible implementation, for the first image patch among multiple image patches, the server uses a feature extraction model to determine multiple attention weights between the multiple image patches and the first image patch, based on the query vector of the first image patch and the key vectors of the multiple image patches.

[0148] For example, the server uses a feature extraction model to perform dot product between the query vector of the first image patch and the key vectors of multiple image patches, obtaining multiple attention scores of the multiple image patches on the first image patch. The server then uses the feature extraction model to normalize these multiple attention scores, obtaining multiple attention weights of the multiple image patches on the first image patch.

[0149] If multiple image patches are divided into three image patches, the query vector of the first image patch is (2, 3, 15), the key vector of the first image patch is (5, 10, 1), the key vector of the second image patch is (5, 1, 2), and the key vector of the third image patch is (6, 2, 0). Then, the server can use a feature extraction model to multiply the query vector (2, 3, 15) of the first image patch with the key vectors of the three image patches (including the first image patch itself) to obtain the attention scores of the three image patches. That is, the attention score of the first image patch itself is (2, 3, 15)·(5, 10, 1) = 55, the attention score of the second image patch on the first image patch is (2, 3, 15)·(5, 1, 2) = 43, and the attention score of the third image patch on the first image patch is (2, 3, 15)·(6, 2, 0) = 18. The server normalizes the three attention scores (55, 43, 18), that is, it normalizes the three attention scores using the Softmax function to obtain the attention weights of the three image patches to the first image patch (0.47, 0.37, 0.16).

[0150] Of course, the above implementation is illustrated by the example of the server determining the attention weight of multiple image blocks to the first image block through a feature extraction model. The method by which the server determines the attention weight of multiple image blocks to other image blocks in the multiple image blocks is the same inventive concept as the method by which multiple image blocks determine the attention weight of multiple image blocks to the first image block, and will not be described in detail here.

[0151] Part Three explains the method by which the server obtains the attention encoding vector of each image patch based on the attention weight and value vector of each image patch using a feature extraction model.

[0152] In one possible implementation, for the first image patch among multiple image patches, the server uses a feature extraction model to perform a weighted summation of the attention weights of the multiple image patches to the first image patch with the value vectors of the multiple image patches to obtain the attention encoding vector of the first image patch.

[0153] Taking three image patches as an example, using the attention weights (0.47, 0.37, 0.16) of the three image patches on the first image patch determined in Part II above, the value vector of the first image patch is (-8, 18, 6), the value vector of the second image patch is (3, 2, 1), and the value vector of the third image patch is (5, 1, 5). The server, through the feature extraction model, based on the attention weights (0.47, 0.37, 0.16) of the three image patches on the first image patch, performs a weighted summation of the value vectors corresponding to the three image patches, that is, 0.47×(-8, 18, 6) + 0.37×(3, 2, 1) + 0.16×(5, 1, 5) = (-1.85, 9.36, 3.99). This vector (-1.85, 9.36, 3.99) is the attention encoding vector of the first image patch, which is also the image patch feature of the first image patch.

[0154] In one possible implementation, the server performs convolutional processing on multiple image patches separately to obtain multiple image patch feature maps, with each feature map corresponding to one image patch. The server then performs fully connected processing on these multiple image patch feature maps to obtain multiple image patch features. In this approach, the feature extraction model is essentially a CNN (Convolutional Neural Networks) model. The server's use of convolutional processing enables rapid feature extraction of image patches, resulting in high efficiency.

[0155] For example, the server inputs multiple image patches into the convolutional layer of the feature extraction model. Each image patch is convolved using at least one convolutional kernel in the convolutional layer, resulting in at least one image patch feature map for each patch. The server then uses the fully connected layer of the feature extraction model to perform fully connected processing on the at least one image patch feature map for each image patch, obtaining the image patch features for each image patch.

[0156] 404. The server performs dimensionality reduction processing on multiple image patch features to obtain multiple dimensionality-reduced features.

[0157] In one possible implementation, the server maps multiple image patch features to a feature space using an image recognition model, obtaining multiple dimensionality-reduced features within the feature space. The dimension of the feature space is lower than the space containing the multiple image patch features. The image recognition model is trained on multiple sample image patches of a sample image and the actual category of the sample image. It has the ability to predict the corresponding image category based on image patch features. The training process of the image recognition model is described in subsequent steps 701-707 and will not be repeated here. In some embodiments, the image recognition model provided in this application is also referred to as a MIL (Multiple-instance Learning) header. This MIL header can be loaded into the feature input of any visual feature extraction model in a plug-and-play manner to implement the image recognition method provided in this application. From the perspective of the MIL header, an image is a "bag," and image patches are multiple instances within the "bag."

[0158] In this implementation, the server can perform dimensionality reduction processing on image patch features using an image recognition model, thereby reducing the dimensionality of the image patch features, decreasing the computational load on the server, and improving the efficiency of image recognition.

[0159] For example, the server uses an image recognition model to perform linear transformations on the features of multiple image patches, obtaining multiple target linear transformation features in the feature space. Each target linear transformation feature corresponds to a dimensionality-reduced feature. The server then uses the image recognition model to perform standardization and activation processing on these multiple target linear transformation features, obtaining multiple dimensionality-reduced features.

[0160] For example, see Figure 2 The server performs the above steps through the feature reduction layer 201 of the image recognition model. That is, the server inputs multiple image patch features into the first linear layer 2011, multiplies each image patch feature by the first linear transformation matrix of the first linear layer 2011, and obtains multiple target linear transformation features in the feature space. The process of multiplying the image patch features by the first linear transformation matrix is ​​also the process of transforming the image patch features to the feature space. After transforming the multiple image patch features to the feature space through the first linear layer 2011, the server performs standardization processing on the multiple target linear transformation features in the feature space through the first standardization layer 2012, that is, transforming the values ​​in the multiple target linear transformation features to conform to a standard normal distribution with a mean of 0 and a variance of 1. The server then performs activation processing on the standardized target linear transformation features through the first activation layer 2013, where activation processing is also a nonlinear transformation.

[0161] In some embodiments, the server performs dimensionality reduction processing on multiple image block features based on the following formula (1) using an image recognition model.

[0162] (1)

[0163] Among them, h i This represents the dimensionality reduction feature of image patch i. `max(A, 0)` is the ReLU activation function of the first normalized layer 2012, which takes the larger value between A and 0. ReLU is an activation function that retains values ​​greater than 0 and adjusts values ​​less than 0 to 0. W1 T Let R be the first linear transformation matrix of the first linear layer 2011. The dimension of the first linear transformation matrix is ​​D×M, where D is the dimension of the image patch features, M is the dimension of the feature space, and R is the dimension of the feature space. M For the feature space, v i Let be the image patch features of image patch i. Norm(x) is also the standardization function of the first standardization layer 2012. E(x) is the expectation of x, which can also be understood as the mean, Var(x) is the variance of x, and γ and β are learnable parameters. It is a random variable.

[0164] 405. The server fuses multiple dimensionality reduction features to obtain the image features of the image.

[0165] In one possible implementation, the server uses an image recognition model to obtain multiple attention weights based on multiple dimensionality reduction features, with each attention weight corresponding to one dimensionality reduction feature. The server then uses the image recognition model to perform a weighted sum of the corresponding dimensionality reduction features based on the multiple attention weights to obtain the image features. Since different image patches contribute differently to the final image recognition result, the attention weights between image patch features can fully utilize these features and amplify the differences between different image patches.

[0166] To provide a clearer explanation of the above embodiments, the following description will be divided into two parts.

[0167] The first part explains the method by which the server uses an image recognition model to perform fully connected processing, standardization, and normalization on multiple dimensionality-reduced features to obtain multiple attention weights.

[0168] In one possible implementation, see Figure 2 The server executes the method described in the first part by using the image feature acquisition layer 202 of the image recognition model 200.

[0169] For example, the server inputs multiple dimensionality reduction features into the second linear layer 2021 of the image feature acquisition layer 202. Using the second linear transformation matrix of the second linear layer 2021, the server performs a linear transformation on the multiple dimensionality reduction features, obtaining multiple first linear transformation features, where each linear transformation feature corresponds to one dimensionality reduction feature. The server then standardizes these multiple first linear transformation features using a second normalization layer 2022. Next, the server activates these standardized first linear transformation features using a second activation layer 2023. Finally, the server performs a third linear transformation on these activated first linear transformation features using a third linear layer 2025, obtaining multiple second linear transformation features. Finally, the server normalizes these second linear transformation features using a normalization layer 2026, resulting in multiple attention weights.

[0170] In some embodiments, the server obtains multiple attention weights based on the following formula (2) through an image recognition model.

[0171] (2)

[0172] Among them, a i For the attention weights of image patch i, W2 T W3 is the second linear transformation matrix. T This is the third linear transformation matrix.

[0173] The second part describes the method by which the server uses an image recognition model to perform a weighted summation of the corresponding dimensionality reduction features based on multiple attention weights to obtain the image features of the image.

[0174] In one possible implementation, see Figure 2 The server executes the method described in the second part by using the image feature acquisition layer 202 of the image recognition model 200.

[0175] For example, the server uses the feature fusion layer 2027 to perform a weighted summation of multiple dimensionality-reduced features and their corresponding attention weights to obtain the image features of the image.

[0176] In some embodiments, the server uses an image recognition model to perform a weighted summation of multiple dimensionality reduction features and their corresponding attention weights based on the following formula (3) to obtain the image features of the image.

[0177] (3)

[0178] Where R represents the image features, N represents the number of image patches, and N is a positive integer.

[0179] 406. The server determines the category of an image based on its image features.

[0180] In one possible implementation, the server classifies image features using an image recognition model to obtain a probability distribution column for the image. The values ​​in the probability distribution column represent the probability that the image corresponds to a category. In response to any value in the probability distribution column representing a probability that meets a target probability condition, the server determines the category corresponding to that value as the image's category.

[0181] For example, the server inputs image features into the classifier of the image recognition model, multiplies the classification matrix of the classifier with the image features, and obtains the probability distribution of the image. In response to any value in the probability distribution representing a probability greater than or equal to a probability threshold, or any value being the largest among multiple values ​​in the probability distribution, the category corresponding to that value is determined as the image category.

[0182] For example, a classifier can determine the category of an image based on image features according to the following formula (4).

[0183] (4)

[0184] Where p is the probability distribution of the image, ω T This is the classification matrix of the classifier.

[0185] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0186] The following will combine Figure 2 , Figure 6 The technical solutions provided in the embodiments of this application are described in conjunction with the optional steps in step 401 above.

[0187] See Figure 6 The server divides the image 601 to be recognized into four image blocks AD, and performs embedding encoding on each of the four image blocks to obtain embedding vectors for each block. These embedding vectors are then input into a feature extraction model (Transformer encoder) to obtain image block features from multiple image blocks. Finally, these image block features are input into an image recognition model 200, which outputs the image category.

[0188] The technical solution provided in this application provides that, during image recognition, multiple image blocks of the image to be recognized are acquired, and feature extraction is performed on each of the multiple image blocks to obtain multiple image block features. When acquiring image features, a method of fusing multiple image block features is adopted. Compared to directly extracting image features, the image features obtained using the technical solution provided in this application have stronger expressive power, and thus, more accurate results can be obtained when determining the image category based on the image features.

[0189] In addition to the image recognition method described in steps 401-406 above, this application embodiment also provides another image recognition method, see [link to relevant documentation]. Figure 7 The methods include:

[0190] 701. The server obtains multiple sample image blocks of the sample image.

[0191] Each sample image corresponds to an actual category. The goal of training the image recognition model is to ensure that when a sample image is input into the trained image recognition model, the predicted category output by the model is the same as the actual category of the sample image.

[0192] Furthermore, the method by which the server obtains multiple sample image blocks of the sample image is the same inventive concept as the method by which the server obtains multiple image blocks of the image to be identified in step 402 above. The implementation process is described in the relevant description of step 402 above, and will not be repeated here.

[0193] 702. The server extracts features from multiple sample image blocks to obtain multiple sample image block features, with one sample image block feature corresponding to one sample image block.

[0194] The process of the server extracting features from multiple sample image blocks is the same inventive concept as the process of extracting features from image blocks in step 403 above. The implementation process is described in the relevant description of step 403 above, and will not be repeated here.

[0195] 703. The server inputs the features of multiple sample image blocks into the image recognition model. The image recognition model then performs dimensionality reduction on the features of the multiple sample image blocks to obtain multiple dimensionality-reduced features.

[0196] The process by which the server performs dimensionality reduction on the features of multiple sample image blocks using an image recognition model is the same inventive concept as the process of the server performing dimensionality reduction on the features of multiple image blocks in step 404 above. The implementation process is described in the relevant description of step 404 above, and will not be repeated here.

[0197] In one possible implementation, multiple sample image patch features are mapped to a feature space to obtain multiple dimensionality-reduced features within the feature space, where the dimension of the feature space is lower than that of the space containing the multiple sample image patch features.

[0198] For example, performing linear transformations on multiple dimensionality-reduced features yields multiple target linear transformation features in the feature space, with each target linear transformation feature corresponding to a dimensionality-reduced feature. Standardizing these multiple target linear transformation features then yields multiple dimensionality-reduced features.

[0199] 704. The server uses an image recognition model to fuse multiple dimensionality reduction features to obtain the sample image features of the sample image.

[0200] The method by which the server fuses multiple dimensionality-reduced features through an image recognition model to obtain sample image features is based on the same inventive concept as the method for obtaining image features in step 405 above. The implementation process is described in the relevant description of step 404 above and will not be repeated here.

[0201] In one possible implementation, multiple attention weights are obtained based on multiple dimensionality reduction features, with each attention weight corresponding to one dimensionality reduction feature. The corresponding dimensionality reduction features are then weighted and summed based on these multiple attention weights to obtain the sample image features of the sample image.

[0202] For example, multiple attention weights are obtained by performing fully connected processing, random deactivation, standardization, and normalization on multiple dimensionality-reduced features. Adding random deactivation during model training can prevent overfitting and improve the model's generalization ability. See also the following examples in some embodiments. Figure 2 Random deactivation is performed by random deactivation layer 2024.

[0203] 705. The server uses an image recognition model to output the first predicted category of the sample image based on the features of the sample image.

[0204] The method by which the server outputs the first predicted category of the sample image based on the features of the sample image through the image recognition model belongs to the same inventive concept as step 406 above. The implementation process is described in the relevant description of step 406 above, and will not be repeated here.

[0205] 706. The server performs fully connected processing and normalization on the features of multiple sample image blocks to obtain the second predicted category of the sample image.

[0206] In one possible implementation, the server performs fully connected processing and normalization processing on multiple sample image patches respectively to obtain image patch prediction types corresponding to each sample image patch. Based on the image patch prediction types corresponding to the multiple sample image patches, the server determines a second prediction type for the sample image.

[0207] For example, the server performs fully connected processing and normalization on multiple sample image patches respectively, obtaining a probability distribution column for each sample image patch. The values ​​in the probability distribution column represent the probability of the corresponding category. The server adds the probability distribution columns of multiple sample images to obtain the probability distribution column of the sample image. In response to any value in the probability distribution column of the sample image meeting the target condition, the category corresponding to that value is determined as the second predicted category of the sample image.

[0208] 707. The server trains the image recognition model based on the first difference information between the first predicted category of the sample image and the actual category of the sample image.

[0209] In one possible implementation, the server trains the image recognition model based on first difference information and second difference information, wherein the second difference information is the difference information between the second predicted category and the actual category.

[0210] For example, the server can train the image recognition model based on the first difference information and the second difference information using the following formula (5).

[0211] (5)

[0212] Where L() is the loss value, y i For the actual category of the sample image, As the first prediction category, For the second prediction category, The weights are used to control the contribution of the first and second difference information to the loss value.

[0213] It should be noted that, in the process of adjusting the model parameters of the image recognition model based on the loss value, the server can use gradient descent, such as batch gradient descent, stochastic gradient descent, Adam, or mini-batch gradient descent. This application does not limit this method.

[0214] The effectiveness of the image recognition model in identifying images of eye diseases was verified. During the verification process, the Transformer model was used as the feature extraction model. Since this application embodiment uses image recognition, the VT (Vision Transformer) model was employed. To pre-train the feature extraction model, a fundus image dataset was collected, containing 345,271 fundus images labeled with five common retinal conditions: 208,733 normal images, 38,284 images of diabetic retinopathy, 21,962 images of age-related macular degeneration, 24,082 images of glaucoma, and 67,230 images of cataracts. Based on this, 95% of the dataset was randomly split for training, and the remaining 5% was used for validation. All fundus images were resized to 384 x 384 pixels for training. ImageNet (image dataset) was used to initialize the model parameters to accelerate convergence.

[0215] For subsequent tasks, two publicly available benchmark databases, APTOS 2019 blind eye detection and the 2020 multi-disease image dataset (RFMiD 2020), were used to validate the effectiveness of the image recognition model provided in this application. APTOS 2019 contains 3662 fundus images for disease grading, with five label categories ranging from 0 to 4, representing increasing disease severity. The RFMiD 2020 dataset contains 1900 images, providing binary disease labels: 0 for normal images and 1 for pathological images. For both datasets, images were divided into five folds for cross-validation and resized to 512 x 512 pixels before being fed into the network. All experiments were performed on a GPU (Graphics Processing Unit) cluster consisting of four NVIDIA Tesls P40 GPUs and 24GB of memory. The Adam optimizer was used to optimize the model parameters, with 16 batches and a maximum training iteration count of 30. The initial learning rate was set to 2 × 10⁻⁶. -4 The learning rate is halved every 2000 iterations. During training, strategies such as random cropping, horizontal flipping, and color jittering were employed to increase the diversity of the training data.

[0216] Referring to Tables 1 and 2, a comparison of the ablation study results of the embodiments of this application on the APTOS 2019 dataset and the RFMiD 2020 dataset is shown. For ease of distinction, VT1 is used below to represent the visual Transformer model using related technologies, which is trained on ImageNet data. VT2 is used to represent the visual Transformer model using related technologies, which is trained on Fundus Image (fundus image) data, i.e., images of eye diseases. MIL-VT is used to represent the combination of the visual Transformer model and the image recognition model provided in the embodiments of this application. MIL is the image recognition model provided in the embodiments of this application, and the training set is also Fundus Image data.

[0217] Table 1

[0218]

[0219] Table 2

[0220]

[0221] AUC stands for Area Under the Curve, where the curve is the Receiver Operating Characteristic Curve (ROC). AUC is used to evaluate the classification performance of a classifier. ACC (Accuracy) is the accuracy, F1 is the F1 metric, and Kappa (cappa) is the Kappa coefficient, used to evaluate classification precision. Recall is the recall rate, and Precision is the precision.

[0222] As can be seen from Tables 1 and 2, using Fundus Image as the training set outperforms using the general Image Net dataset. Furthermore, the performance is further improved by employing the image recognition model (MIL) provided in this application.

[0223] After ablation learning, the technical solution provided in this application embodiment is compared with the technical solutions of other models on the APTOS 2019 dataset and the RFMiD 2020 dataset. The results are shown in Tables 3 and 4.

[0224] Table 3

[0225]

[0226] Table 4

[0227]

[0228] Models 1 and 2 both use the ResNet34 (Residual Network-34) architecture, but Model 1 is trained on the ImageNet dataset, while Model 2 is trained on the Fundus Image dataset. Model 3 is a (Diabetic Retinopathy Detection Through Integration of DeepLearning Classication Framework). Model 4 is CANet (Cross-disease Attention Network). Model 5 is GREEN-ResNet50 (Graph Residual Re-ranking Network-50). Model 6 is GREEN-SE-ResNet50 (Graph Residual Re-ranking Network-50-Squeeze and Excitation).

[0229] As can be seen from Tables 3 and 4, the MIL-VT provided by the embodiments of this application has better performance than the technical solutions in related technologies.

[0230] The technical solution provided in this application, when training an image recognition model, first acquires multiple sample image blocks of a sample image, and then extracts features from each of these blocks to obtain multiple sample image block features. When acquiring the sample image features, a method of fusing the features of multiple sample image blocks is used. Compared to directly extracting the sample image features, the sample image features obtained using the technical solution provided in this application have stronger expressive power, and the image recognition model trained based on these sample image features thus has higher recognition accuracy.

[0231] Figure 8 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application. See also... Figure 8 The device includes: an image patch acquisition module 801, a first feature extraction module 802, a dimensionality reduction module 803, a feature fusion module 804, and a category determination module 805.

[0232] The image block acquisition module 801 is used to acquire multiple image blocks of the image to be identified.

[0233] The first feature extraction module 802 is used to extract features from multiple image blocks to obtain multiple image block features, with one image block feature corresponding to one image block.

[0234] The dimensionality reduction module 803 is used to perform dimensionality reduction processing on multiple image patch features to obtain multiple dimensionality-reduced features.

[0235] The feature fusion module 804 is used to fuse multiple dimensionality-reduced features to obtain the image features of the image.

[0236] The category determination module 805 is used to determine the category of an image based on its features.

[0237] In one possible implementation, the first feature extraction module 802 is used to encode multiple image blocks separately based on an attention mechanism to obtain multiple image block features.

[0238] In one possible implementation, the dimensionality reduction module 803 is used to map multiple image block features to a feature space through an image recognition model, thereby obtaining multiple dimensionality-reduced features in the feature space, wherein the dimension of the feature space is lower than that of the space containing the multiple image block features.

[0239] In one possible implementation, the dimensionality reduction module 803 is used to perform linear transformations on multiple image patch features respectively, obtaining multiple target linear transformation features in the feature space, where each target linear transformation feature corresponds to a dimensionality reduction feature. The multiple target linear transformation features are then subjected to standardization and activation processing to obtain multiple dimensionality reduction features.

[0240] In one possible implementation, the feature fusion module 804 is used to obtain multiple attention weights based on multiple dimensionality reduction features, where one attention weight corresponds to one dimensionality reduction feature.

[0241] Based on multiple attention weights, the corresponding dimensionality reduction features are weighted and summed to obtain the image features of the image.

[0242] In one possible implementation, the feature fusion module 804 is used to perform fully connected processing, standardization processing and normalization processing on multiple dimensionality-reduced features to obtain multiple attention weights.

[0243] In one possible implementation, the category determination module 805 is used to classify image features to obtain a probability distribution column of the image, wherein the values ​​in the probability distribution column are used to represent the probability that the image corresponds to a category.

[0244] In response to any value in the probability distribution series representing a probability that meets the target probability condition, the category corresponding to any value is determined as the category of the image.

[0245] It should be noted that the image recognition device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image recognition device and the image recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0246] The technical solution provided in this application provides that, during image recognition, multiple image blocks of the image to be recognized are acquired, and feature extraction is performed on each of the multiple image blocks to obtain multiple image block features. When acquiring image features, a method of fusing multiple image block features is adopted. Compared to directly extracting image features, the image features obtained using the technical solution provided in this application have stronger expressive power, and thus, more accurate results can be obtained when determining the image category based on the image features.

[0247] Figure 9 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application. See also... Figure 9 The device includes: a sample image block acquisition module 901, a second feature extraction module 902, an input module 903, and a training module 904.

[0248] The sample image block acquisition module 901 is used to acquire multiple sample image blocks of a sample image.

[0249] The second feature extraction module 902 is used to extract features from multiple sample image blocks to obtain multiple sample image block features, with one sample image block feature corresponding to one sample image block.

[0250] Input module 903 is used to input multiple sample image patch features into the image recognition model. The image recognition model performs dimensionality reduction on these features to obtain multiple dimensionality-reduced features. These dimensionality-reduced features are then fused to obtain the sample image features of the sample image. Based on these sample image features, the first predicted category of the sample image is output.

[0251] Training module 904 is used to train the image recognition model based on the first difference information between the first predicted category of the sample image and the actual category of the sample image.

[0252] In one possible implementation, the input module 903 is used to map the features of multiple sample image blocks to a feature space to obtain multiple dimensionality-reduced features in the feature space, wherein the dimension of the feature space is lower than that of the space in which the features of the multiple sample image blocks reside.

[0253] In one possible implementation, the input module 903 is used to perform linear transformations on multiple dimensionality reduction features to obtain multiple target linear transformation features in the feature space, where one target linear transformation feature corresponds to one dimensionality reduction feature.

[0254] By standardizing the linear transformation features of multiple targets, multiple dimensionality-reduced features are obtained.

[0255] In one possible implementation, the input module 903 is used to obtain multiple attention weights based on multiple dimensionality reduction features, where each attention weight corresponds to one dimensionality reduction feature. Based on the multiple attention weights, the corresponding dimensionality reduction features are weighted and summed to obtain the sample image features of the sample image.

[0256] In one possible implementation, the input module 903 is used to perform fully connected processing, random deactivation processing, standardization processing and normalization processing on multiple dimensionality-reduced features to obtain multiple attention weights.

[0257] In one possible implementation, the device further includes:

[0258] The prediction category determination module is used to perform fully connected processing and normalization processing on the features of multiple sample image blocks to obtain the second predicted category of the sample image.

[0259] The training module 904 is used to train the image recognition model based on the first difference information and the second difference information, where the second difference information is the difference information between the second predicted category and the actual category.

[0260] It should be noted that the image recognition device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image recognition device and the image recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0261] The technical solution provided in this application, when training an image recognition model, first acquires multiple sample image patches from a sample image, and then extracts features from each of these patches to obtain multiple sample image patch features. When acquiring the sample image features, a method of fusing the features of multiple sample image patches is adopted. Compared to directly extracting the sample image features, the sample image features obtained using the technical solution provided in this application have stronger expressive power, and thus, more accurate results can be obtained when determining the image category based on these image features.

[0262] This application provides a computer device for performing the above-described method. This computer device can be implemented as a terminal or a server. The structure of the terminal will be described below:

[0263] Figure 10 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal 1000 can be a smartphone, tablet computer, laptop computer, or desktop computer. The terminal 1000 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0264] Typically, terminal 1000 includes one or more processors 1001 and one or more memories 1002.

[0265] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0266] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one computer program, which is executed by the processor 1001 to implement the image recognition method provided in the method embodiments of this application.

[0267] In some embodiments, the terminal 1000 may also optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, memory 1002, and peripheral device interface 1003 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1003 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1009.

[0268] Peripheral device interface 1003 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1001 and memory 1002. In some embodiments, processor 1001, memory 1002 and peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1001, memory 1002 and peripheral device interface 1003 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0269] The radio frequency (RF) circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1004 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc.

[0270] Display screen 1005 is used to display a user interface (UI). This UI may include graphics, text, icons, video, and any combination thereof. When display screen 1005 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1001 for processing. In this case, display screen 1005 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard.

[0271] The camera assembly 1006 is used to capture images or videos. Optionally, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal.

[0272] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 1001 for processing, or input to the radio frequency circuit 1004 to realize voice communication.

[0273] The power supply 1009 is used to supply power to the various components in the terminal 1000. The power supply 1009 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery.

[0274] In some embodiments, the terminal 1000 further includes one or more sensors 1010. The one or more sensors 1010 include, but are not limited to: an acceleration sensor 1011, a gyroscope sensor 1012, a pressure sensor 1013, an optical sensor 1015, and a proximity sensor 1016.

[0275] Accelerometer 1011 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established with terminal 1000.

[0276] The gyroscope sensor 1012 can detect the orientation and rotation angle of the terminal 1000. The gyroscope sensor 1012 can work in conjunction with the accelerometer sensor 1011 to collect the user's 3D movements on the terminal 1000.

[0277] The pressure sensor 1013 can be installed on the side bezel of the terminal 1000 and / or on the lower layer of the display screen 1005. When the pressure sensor 1013 is installed on the side bezel of the terminal 1000, it can detect the user's grip signal on the terminal 1000, and the processor 1001 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1013. When the pressure sensor 1013 is installed on the lower layer of the display screen 1005, the processor 1001 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1005.

[0278] An optical sensor 1015 is used to collect ambient light intensity. In one embodiment, a processor 1001 can control the display brightness of a display screen 1005 based on the ambient light intensity collected by the optical sensor 1015.

[0279] The proximity sensor 1016 is used to detect the distance between the user and the front of the terminal 1000.

[0280] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on terminal 1000 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0281] The aforementioned computer equipment can also be implemented as a server. The structure of a server is described below:

[0282] Figure 11 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1100 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1101 and one or more memories 1102. The one or more memories 1102 store at least one computer program, which is loaded and executed by the one or more processors 1101 to implement the methods provided in the various method embodiments described above. Of course, the server 1100 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1100 may also include other components for implementing device functions, which will not be elaborated upon here.

[0283] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the image recognition method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0284] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the image recognition method described above.

[0285] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0286] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0287] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image recognition method, characterized in that, The method includes: Obtain multiple image patches of the image to be identified; Using a Transformer encoder, features are extracted from the multiple image patches to obtain multiple image patch features, where each image patch feature corresponds to one image patch; The image recognition model performs linear transformations on the features of the multiple image patches to obtain multiple target linear transformation features in the feature space, with each target linear transformation feature corresponding to a dimensionality reduction feature. The multiple target linear transformation features are then standardized and activated to obtain multiple dimensionality reduction features. The dimension of the feature space is lower than the space containing the multiple image patch features. The image recognition model can be loaded into the feature input of any visual feature extraction model in a plug-and-play manner. The image features of the image are obtained by fusing the multiple dimensionality reduction features using the image recognition model. The image category is determined based on the image features using the image recognition model.

2. The method according to claim 1, characterized in that, The step of extracting features from the plurality of image patches to obtain the features of the plurality of image patches includes: Based on the attention mechanism, the multiple image patches are encoded respectively to obtain the features of the multiple image patches.

3. The method according to claim 1, characterized in that, The process of fusing the multiple dimensionality reduction features to obtain the image features of the image includes: Based on the multiple dimensionality reduction features, multiple attention weights are obtained, and each attention weight corresponds to one of the dimensionality reduction features; Based on the multiple attention weights, the corresponding dimensionality reduction features are weighted and summed to obtain the image features of the image.

4. The method according to claim 3, characterized in that, The process of obtaining multiple attention weights based on the multiple dimensionality reduction features includes: The multiple dimensionality-reduced features are subjected to fully connected processing, standardization processing, and normalization processing to obtain the multiple attention weights.

5. The method according to claim 1, characterized in that, Determining the category of the image based on the image features includes: The image features are classified to obtain a probability distribution column of the image, and the values ​​in the probability distribution column are used to represent the probability that the image corresponds to a category; In response to any value in the probability distribution column representing a probability that meets the target probability condition, the category corresponding to the given value is determined as the category of the image.

6. An image recognition method, characterized in that, The method includes: Obtain multiple sample image patches from the sample image; Using a Transformer encoder, feature extraction is performed on the multiple sample image blocks to obtain multiple sample image block features, where one sample image block feature corresponds to one sample image block; The multiple sample image patch features are input into an image recognition model. The model performs linear transformations on each of the sample image patch features to obtain multiple target linear transformation features in a feature space, where each target linear transformation feature corresponds to a dimensionality reduction feature. These target linear transformation features are then standardized and activated to obtain multiple dimensionality reduction features, where the dimension of the feature space is lower than the dimension of the sample image patch features. These multiple dimensionality reduction features are then fused to obtain the sample image features of the sample image. Based on these sample image features, the first predicted category of the sample image is output. The image recognition model is trained based on the first difference information between the first predicted category of the sample image and the actual category of the sample image; The image recognition model can be loaded onto the feature input of any visual feature extraction model in a plug-and-play manner.

7. The method according to claim 6, characterized in that, The process of fusing the multiple dimensionality reduction features to obtain the sample image features of the sample image includes: Based on the multiple dimensionality reduction features, multiple attention weights are obtained, and each attention weight corresponds to one of the dimensionality reduction features; Based on the multiple attention weights, the corresponding dimensionality reduction features are weighted and summed to obtain the sample image features of the sample image.

8. The method according to claim 7, characterized in that, The process of obtaining multiple attention weights based on the multiple dimensionality reduction features includes: The multiple dimensionality-reduced features are subjected to fully connected processing, random deactivation processing, standardization processing, and normalization processing to obtain the multiple attention weights.

9. The method according to claim 6, characterized in that, Before training the image recognition model based on the first difference information between the first predicted category and the actual category of the sample image, the method further includes: The features of the multiple sample image blocks are subjected to fully connected processing and normalization processing to obtain the second predicted category of the sample image; Training the image recognition model based on the first difference information between the first predicted category and the actual category of the sample image includes: The image recognition model is trained based on the first difference information and the second difference information, wherein the second difference information is the difference information between the second predicted category and the actual category.

10. An image recognition device, characterized in that, The device includes: The image patch acquisition module is used to acquire multiple image patches of the image to be identified; The first feature extraction module is used to extract features from the plurality of image blocks using a Transformer encoder to obtain a plurality of image block features, wherein one image block feature corresponds to one image block; The dimensionality reduction module is used to perform linear transformations on the features of the multiple image blocks using an image recognition model to obtain multiple target linear transformation features in the feature space, where each target linear transformation feature corresponds to a dimensionality reduction feature; and to perform standardization and activation processing on the multiple target linear transformation features to obtain multiple dimensionality reduction features. The feature fusion module is used to fuse the multiple dimensionality-reduced features to obtain the image features of the image. The dimension of the feature space is lower than the space where the multiple image patch features are located. The image recognition model can be loaded onto the feature input of any visual feature extraction model in a plug-and-play manner. The category determination module is used to determine the category of the image based on the image features using the image recognition model.

11. The apparatus according to claim 10, characterized in that, The first feature extraction module is used to encode the multiple image blocks respectively based on an attention mechanism to obtain the features of the multiple image blocks.

12. The apparatus according to claim 10, characterized in that, The feature fusion module is used for: Based on the multiple dimensionality reduction features, multiple attention weights are obtained, and each attention weight corresponds to one of the dimensionality reduction features; Based on the multiple attention weights, the corresponding dimensionality reduction features are weighted and summed to obtain the image features of the image.

13. The apparatus according to claim 12, characterized in that, The feature fusion module is used to perform fully connected processing, standardization processing, and normalization processing on the multiple dimensionality-reduced features to obtain the multiple attention weights.

14. The apparatus according to claim 10, characterized in that, The category determination module is used for: The image features are classified to obtain a probability distribution column of the image, and the values ​​in the probability distribution column are used to represent the probability that the image corresponds to a category; In response to any value in the probability distribution column representing a probability that meets the target probability condition, the category corresponding to the given value is determined as the category of the image.

15. An image recognition device, characterized in that, The device includes: The sample image patch acquisition module is used to acquire multiple sample image patches from a sample image. The second feature extraction module is used to extract features from the plurality of sample image blocks using a Transformer encoder to obtain multiple sample image block features, wherein one sample image block feature corresponds to one sample image block; An input module is used to input the features of the multiple sample image patches into an image recognition model. The image recognition model performs linear transformations on the features of the multiple sample image patches to obtain multiple target linear transformation features in a feature space, where each target linear transformation feature corresponds to a dimensionality reduction feature. The multiple target linear transformation features are then standardized and activated to obtain multiple dimensionality reduction features, where the dimension of the feature space is lower than the dimension of the space containing the multiple sample image patch features. The multiple dimensionality reduction features are then fused to obtain the sample image features of the sample image. Based on the sample image features, a first predicted category of the sample image is output. The training module is used to train the image recognition model based on the first difference information between the first predicted category of the sample image and the actual category of the sample image; The image recognition model can be loaded onto the feature input of any visual feature extraction model in a plug-and-play manner.

16. The apparatus according to claim 15, characterized in that, The input module is used for: Based on the multiple dimensionality reduction features, multiple attention weights are obtained, and each attention weight corresponds to one of the dimensionality reduction features; Based on the multiple attention weights, the corresponding dimensionality reduction features are weighted and summed to obtain the sample image features of the sample image.

17. The apparatus according to claim 16, characterized in that, The input module is used for: The multiple dimensionality-reduced features are subjected to fully connected processing, random deactivation processing, standardization processing, and normalization processing to obtain the multiple attention weights.

18. The apparatus according to claim 15, characterized in that, The device further includes: The prediction category determination module is used to perform fully connected processing and normalization processing on the features of the multiple sample image blocks to obtain the second predicted category of the sample image; The training module is used to train the image recognition model based on the first difference information and the second difference information, wherein the second difference information is the difference information between the second predicted category and the actual category.

19. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the image recognition method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the image recognition method as described in any one of claims 1 to 9.

21. A computer program product comprising program code stored in a computer-readable storage medium, wherein a processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform an image recognition method as described in any one of claims 1 to 9.

Images