Image recognition method, device, equipment and storage medium

Abstract

The application discloses an image recognition method, device and equipment and a storage medium, and belongs to the technical field of computers. The technical scheme provided by the embodiment of the application divides a target three-dimensional image into multiple sub-images when the target three-dimensional image is recognized, so that the dimensionality of the target three-dimensional image is reduced. Time attention features and space attention features of the target three-dimensional image are obtained based on the multiple sub-images, and image features of the target three-dimensional image are obtained based on the time attention features and the space attention features. In the feature extraction process, a three-dimensional convolution kernel is not required, and the obtained image features have strong expression ability. Based on the image features, the target three-dimensional image can be classified, so that the efficiency of image recognition is improved while the accuracy of image recognition is ensured.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image recognition method, apparatus, device, and storage medium. Background Technology

[0002] With the development of computer technology, artificial intelligence technology is booming. As a branch of artificial intelligence, image recognition technology is being applied in an increasingly wide range of applications. For example, image recognition technology can be applied to facial recognition scenarios, where image recognition models are used to identify images containing faces to obtain the identity information corresponding to the faces; or it can be applied to medical scenarios, where image recognition models are used to identify medical images to obtain the disease types corresponding to the medical images.

[0003] In related technologies, for some three-dimensional images, image recognition models often use three-dimensional convolution kernels to process the three-dimensional images in order to obtain the corresponding category of the three-dimensional image. For example, in medical scenarios, three-dimensional convolution kernels are used to process three-dimensional CT (Computed Tomography) images to obtain the disease type corresponding to the three-dimensional CT image.

[0004] However, the image recognition model has a high computational cost when using 3D convolution to process 3D images, resulting in low efficiency. Summary of the Invention

[0005] This application provides an image recognition method, apparatus, device, and storage medium, which can improve the efficiency of image recognition while ensuring the image recognition effect. The technical solution is as follows:

[0006] On the one hand, an image recognition method is provided, the method comprising:

[0007] Obtain multiple sub-images of the target 3D image;

[0008] Based on the multiple sub-images, multiple temporal attention features and multiple spatial attention features of the target 3D image are obtained. The temporal attention features are used to represent the importance of the same position in different sub-images, and the spatial attention features are used to represent the importance of different positions in each sub-image.

[0009] The multiple temporal attention features and the multiple spatial attention features are fused to obtain the image features of the target 3D image;

[0010] Based on the image features, the category of the target 3D image is determined.

[0011] In one possible implementation, encoding the plurality of image feature groups based on an attention mechanism to obtain spatial attention features for each of the image feature groups includes:

[0012] Obtain the key matrix, query matrix, and value matrix of each image block feature in each of the image feature groups;

[0013] Based on the key matrix, query matrix, and value matrix of each image block feature in each image feature group, the spatial attention features of each image feature group are obtained.

[0014] On the one hand, an image recognition method is provided, the method comprising:

[0015] Obtain multiple sample sub-images of the sample 3D image;

[0016] The multiple sample sub-images are input into an image recognition model. Based on these sub-images, the image recognition model obtains multiple temporal attention features and multiple spatial attention features of the sample 3D image. The temporal attention features are fusion features at corresponding positions in the multiple sample sub-images, and the spatial attention features are fusion features at different positions in each of the sample sub-images. The multiple temporal attention features and the multiple spatial attention features are fused to obtain the image features of the sample 3D image. Based on these image features, the predicted category of the sample 3D image is output.

[0017] The model parameters of the image recognition model are adjusted based on the difference information between the predicted category and the labeled category of the sample 3D image.

[0018] On one hand, an image recognition device is provided, the device comprising:

[0019] The first sub-image acquisition module is used to acquire multiple sub-images of the target 3D image;

[0020] The first feature acquisition module is used to acquire multiple temporal attention features and multiple spatial attention features of the target 3D image based on the multiple sub-images. The temporal attention features are used to represent the importance of the same position in different sub-images, and the spatial attention features are used to represent the importance of different positions in each sub-image.

[0021] The first feature fusion module is used to fuse the multiple temporal attention features and the multiple spatial attention features to obtain the image features of the target three-dimensional image;

[0022] The first category determination module is used to determine the category of the target 3D image based on the image features.

[0023] In one possible implementation, the first sub-image acquisition module is used to divide the target three-dimensional image into multiple slice images; and acquire the multiple sub-images based on the multiple slice images, wherein each sub-image includes at least one of the slice images.

[0024] In one possible implementation, the first sub-image acquisition module is configured to combine every three adjacent slice images from the plurality of slice images into a sub-image, wherein every two adjacent sub-images in the plurality of sub-images include one or two identical slice images.

[0025] In one possible implementation, the first feature acquisition module is used to extract features from the plurality of sub-images respectively to obtain sub-image feature groups for each of the sub-images; and based on the sub-image feature groups of the plurality of sub-images, to acquire a plurality of temporal attention features and a plurality of spatial attention features of the target 3D image.

[0026] In one possible implementation, the first feature acquisition module is used to divide any one of the plurality of sub-images into a plurality of image blocks; to extract features from each of the plurality of image blocks to obtain image block features of each of the image blocks, and the image block features of the plurality of image blocks constitute a sub-image feature group of the sub-image.

[0027] In one possible implementation, the first feature acquisition module is configured to acquire multiple temporal feature sequences from the sub-image feature groups of the multiple sub-images, the temporal feature sequences including image block features of corresponding image blocks in each of the sub-image feature groups; encode the multiple temporal feature sequences based on an attention mechanism to obtain temporal attention features of each of the temporal feature sequences; and encode the image feature groups of the multiple sub-images based on an attention mechanism to obtain spatial attention features of each of the image feature groups.

[0028] In one possible implementation, the first feature acquisition module is used to acquire the key matrix, query matrix, and value matrix of each image block feature in each of the time feature sequences; and to acquire the time attention features of each of the time feature sequences based on the key matrix, query matrix, and value matrix of each image block feature in each of the time feature sequences.

[0029] In one possible implementation, the first feature acquisition module is configured to acquire multiple temporal feature sequences from sub-image feature groups of the multiple sub-images, the temporal feature sequences including image block features corresponding to image blocks in each sub-image feature group; encode the multiple temporal feature sequences based on an attention mechanism to obtain temporal attention features of each temporal feature sequence; fuse each sub-image feature group and the temporal attention features of each temporal feature sequence to obtain a fused feature group of each sub-image; and encode the fused feature group of each sub-image based on an attention mechanism to obtain spatial attention features of each sub-image.

[0030] In one possible implementation, the first feature acquisition module is used to acquire the key matrix, query matrix, and value matrix of each image block feature in each of the image feature groups; and to acquire the spatial attention features of each of the image feature groups based on the key matrix, query matrix, and value matrix of each image block feature in each of the image feature groups.

[0031] In one possible implementation, the first category determination module is used to perform fully connected processing and activation processing on the image features to obtain a probability distribution column of the target 3D image; in response to any value in the probability distribution column meeting the target probability condition, the category corresponding to the value is determined as the category of the target 3D image.

[0032] In one possible implementation, the first feature acquisition module is used to encode the plurality of sub-images using a Transformer to obtain a plurality of temporal attention features and a plurality of spatial attention features of the target 3D image.

[0033] On one hand, an image recognition device is provided, the device comprising:

[0034] The second sub-image acquisition module is used to acquire multiple sample sub-images of the sample 3D image;

[0035] The second category determination module is used to input the multiple sample sub-images into an image recognition model, and through the image recognition model, obtain multiple temporal attention features and multiple spatial attention features of the sample 3D image based on the multiple sample sub-images. The temporal attention features are used to represent the importance of the same position in different sub-images, and the spatial attention features are used to represent the importance of different positions in each sub-image. The multiple temporal attention features and the multiple spatial attention features are fused to obtain the image features of the sample 3D image. Based on the image features, the predicted category of the sample 3D image is output.

[0036] The adjustment module is used to adjust the model parameters of the image recognition model based on the difference information between the predicted category and the labeled category of the sample 3D image.

[0037] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the image recognition method.

[0038] On one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer program, which is loaded and executed by a processor to implement the image recognition method.

[0039] On one hand, a computer program product or computer program is provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the image recognition method described above.

[0040] The technical solution provided in this application divides a target 3D image into multiple sub-images during recognition, thereby achieving dimensionality reduction. Temporal and spatial attention features of the target 3D image are obtained based on these sub-images. Image features of the target 3D image are then extracted based on these temporal and spatial attention features. The feature extraction process does not require the participation of a 3D convolution kernel, resulting in image features with strong expressive power. Classifying the target 3D image based on these image features improves the efficiency of image recognition while ensuring accuracy. Attached Figure Description

[0041] To illustrate the technical solutions in the embodiments of this application more clearly, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the implementation environment of an image recognition method provided in an embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the structure of an image recognition model provided in an embodiment of this application;

[0044] Figure 3This is a flowchart of an image recognition method provided in an embodiment of this application;

[0045] Figure 4 This is a flowchart of an image recognition method provided in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of an interface provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram illustrating an embodiment of obtaining a sub-image provided in this application;

[0048] Figure 7 This is a schematic diagram of a time feature sequence provided in an embodiment of this application;

[0049] Figure 8 This is a flowchart of an image recognition method provided in an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application;

[0051] Figure 10 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application;

[0052] Figure 11 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0053] Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0055] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0056] In this application, the term "at least one" means one or more, and "multiple" means two or more, for example, multiple reference face images means two or more reference face images.

[0057] Artificial intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0058] Computer vision (CV) is a science that studies how to make machines "see". More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes to identify, track and measure targets, and further performs image processing to make the computer-processed images more suitable for human observation or transmission to instruments for detection.

[0059] The technical solutions provided in this application can also be combined with cloud technology, for example, by deploying the trained image recognition model on a cloud server. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0060] Among cloud technologies, medical cloud refers to the creation of a healthcare service cloud platform using cloud computing, combining medical technologies with new technologies such as cloud computing, mobile technology, multimedia, 4G communication, big data, and the Internet of Things. This enables the sharing of medical resources and expands the scope of medical care. Because of the application of cloud computing technology, medical cloud improves the efficiency of medical institutions and facilitates access to medical care for residents. For example, hospital appointment registration, electronic medical records, and medical insurance are all products of the integration of cloud computing and the medical field. Medical cloud also has advantages such as data security, information sharing, dynamic expansion, and global deployment. For instance, the image recognition model provided in this application embodiment is deployed on a healthcare service cloud platform.

[0061] Transformer: A neural network based on its own attention mechanism, widely used in speech recognition, image recognition, natural language processing, and other fields. It generates a new output sequence by adding weights to the input sequence through an attention mechanism one or more times, weakening or forgetting content that does not conform to the attention model.

[0062] Normalization: Mapping sequences with different value ranges to the interval (0, 1) to facilitate data processing. In some cases, normalized values ​​can be directly expressed as probabilities.

[0063] Learning rate: Used to control the learning progress of the model. The learning rate guides the model in adjusting network weights using the gradient of the loss function during gradient descent. If the learning rate is too large, the loss function may directly skip the global optimum, resulting in excessive loss. If the learning rate is too small, the loss function changes very slowly, greatly increasing the convergence complexity of the network and making it easy to get trapped in local minima or saddle points.

[0064] Embedded coding, mathematically speaking, represents a correspondence, that is, mapping data in space X to space Y using a function F. This function F is injective, and the mapping result preserves the structure. An injective function means that the mapped data uniquely corresponds to the original data, and preserving the structure means that the order of the original data remains the same. For example, if there are data X1 and X2 before mapping, after mapping we get Y1 corresponding to X1 and Y2 corresponding to X2. If the original data X1 > X2, then correspondingly, the mapped data Y1 > Y2. For words, this means mapping words to another space to facilitate subsequent machine learning and processing.

[0065] Attention weights represent the importance of a piece of data during training or prediction. Importance indicates the magnitude of the influence of input data on output data. Data with high importance corresponds to higher attention weights, while data with low importance corresponds to lower attention weights. The importance of data varies in different scenarios, and training the model to assign attention weights is essentially the process of determining data importance.

[0066] Optionally, the computer device provided in this application embodiment can be implemented as a terminal or a server. The implementation environment consisting of a terminal and a server will be described below.

[0067] Figure 1 This is a schematic diagram illustrating the implementation environment of an image recognition method provided in this application embodiment. See also... Figure 1 The implementation environment may include terminal 110 and server 140.

[0068] Terminal 110 is connected to server 140 via a wireless or wired network. Optionally, terminal 110 may be a smartphone, tablet, laptop, desktop computer, smartwatch, etc., but is not limited to these. Terminal 110 has an application that supports image recognition installed and running.

[0069] Optionally, server 140 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. In some embodiments, a user can upload an image to be recognized to server 140 via terminal 110, and server 140 will perform the image recognition method provided in this application embodiment on the uploaded image. After recognition is completed, server 140 will send the recognition result to terminal 110, and the user can view the image recognition result through terminal 110.

[0070] Optionally, terminal 110 generally refers to one of a plurality of terminals, and this application embodiment only uses terminal 110 as an example for illustration.

[0071] Those skilled in the art will understand that the number of terminals 110 described above can be more or less. For example, there may be only one terminal 110, or there may be dozens or hundreds of terminals 110, or even more. In this case, other terminals are also included in the above implementation environment. The embodiments of this application do not limit the number of terminals or the type of devices.

[0072] After introducing the implementation environment of the image recognition method provided in the embodiments of this application, the application scenarios of the image recognition method provided in the embodiments of this application will be described below. It should be noted that the terminal in the following description is the terminal 110 in the above implementation environment, and the server is the server 140 in the above implementation environment.

[0073] The image recognition method provided in this application can be applied to various image recognition scenarios. For example, it can be applied to the recognition of three-dimensional lung images, three-dimensional building images, three-dimensional vehicle images, or three-dimensional material images, etc. This application does not limit the application to these scenarios.

[0074] In scenarios involving the recognition of three-dimensional lung images, such as determining whether a three-dimensional lung image represents COVID-19 or ordinary pneumonia, or whether it represents ordinary pneumonia or normal lungs, this application embodiment does not impose limitations. Since identifying the disease category corresponding to a three-dimensional lung image requires considerable experience, and disease category identification is the foundation of effective treatment, less experienced medical personnel can utilize the image recognition method provided in this application embodiment to identify the three-dimensional lung image, determine its type, and use this type as a reference for diagnosis, thereby improving the accuracy of lung disease identification. When three-dimensional lung image recognition is required, medical personnel upload the three-dimensional lung image to the server via a terminal. The server then executes the image recognition method provided in this application embodiment on the received three-dimensional lung image to determine its category. The category of the three-dimensional lung image indicates whether it represents COVID-19 or ordinary pneumonia. The server returns the category of the 3D lung image to the terminal, which then displays the category to medical staff. The category indicates whether the 3D lung image represents COVID-19, ordinary pneumonia, or normal lungs. This category can serve as a reference for medical staff in disease identification, thereby assisting them in formulating treatment plans.

[0075] In scenarios involving the recognition of 3D material images, such as identifying whether a 3D material image contains cracks, the 3D material image is typically captured by an AFM (Atomic Force Microscope). When a crack exists in a material, the impact of external force is concentrated at the crack location, leading to crack propagation and material failure. This aging can be particularly severe when the material is used as a structural support. However, the initial degree of cracking is often small, making it difficult for the human eye to effectively identify. In such cases, the user can upload the 3D material image to a server via a terminal. The server then executes the image recognition method provided in this application to determine the category of the 3D material image. The category indicates whether the 3D material image contains cracks or not. The server returns the category to the terminal, which then displays the category to the user, allowing the user to determine whether the metal 3D material image contains cracks.

[0076] In scenarios involving the recognition of 3D building images, such as identifying the presence of a specific type of building within an image, consider a set of 3D building images. This set might include buildings of three types: Type A, Type B, and Type C. A user can upload these images to a server via a terminal. The server then applies the image recognition method provided in this embodiment to determine the type of each 3D building image within the set. The type of a 3D building image indicates whether it contains Type A, Type B, or Type C buildings. The server then displays the type of the set of 3D building images to the user via the terminal, allowing the user to determine the type of each 3D building image within the set.

[0077] In scenarios involving the recognition of 3D vehicle images, such as identifying the presence of a specific type of vehicle within an image, consider a set of 3D vehicle images. This set might include vehicles of three types: D, E, and F. A user can upload this set of 3D vehicle images to a server via a terminal. The server then applies the image recognition method provided in this embodiment to determine the type of each 3D vehicle image in the set. The type of a 3D vehicle image indicates whether it contains type D vehicles, type E vehicles, or type F vehicles. The server then displays the type of the set of 3D vehicle images to the user via the terminal, allowing the user to determine the type of each 3D vehicle image within the set.

[0078] It should be noted that the above description of the application scenario is based on the example of the terminal acquiring a target 3D image, uploading the target 3D image to the server, and the server recognizing the target 3D image. In other possible implementations, after the terminal acquires the target 3D image, it can also directly recognize the target 3D image without uploading the target 3D image to the server. This application embodiment does not limit this.

[0079] In this embodiment of the application, a computer device can implement the image recognition method provided in this embodiment of the application by means of an image recognition model. The following will be combined with... Figure 2 The structure of the image recognition model provided in the embodiments of this application will be described.

[0080] See Figure 2 The image recognition model 200 includes a feature extraction layer 201, a temporal attention feature acquisition layer 202, a spatial attention feature acquisition layer 203, and an image recognition layer 204. It should be noted that the connection methods of the various layers in the image recognition model can be as follows: Figure 2 Any of the methods shown.

[0081] The feature extraction layer 201 extracts features from multiple sub-images of the target 3D image to obtain sub-image feature groups for each sub-image. The temporal attention feature acquisition layer 202 acquires the temporal attention features of the target 3D image based on the sub-image feature groups of multiple sub-images. The spatial attention feature acquisition layer 203 acquires the spatial attention features of the target 3D image based on the sub-image feature groups of multiple sub-images. The image recognition layer 204 performs recognition based on the image features of the target 3D image to determine the category of the target 3D image.

[0082] In some embodiments, the feature extraction layer 201 is a convolutional layer used to perform convolution processing on multiple image blocks of each sub-image to obtain image block features of each image block, and the image block features of each image block constitute the sub-image feature group of each image block.

[0083] In some embodiments, the temporal attention feature acquisition layer 202 is a Transformer encoder. The spatial attention feature acquisition layer 203 is a Transformer encoder.

[0084] In some embodiments, the image recognition layer 204 is a classifier. After inputting the image features of the target 3D image into the image recognition layer 204, the probability distribution column of the target 3D image can be obtained. Each number in the distribution column is the probability of the target 3D image corresponding to a category.

[0085] It should be noted that, Figure 2 The structure of the image recognition model shown is merely an example. In other possible implementations, the image recognition model may have other structures, and this application does not limit this.

[0086] After introducing the implementation environment, application scenarios, and structure of the image recognition model in the embodiments of this application, the image recognition method provided in the embodiments of this application will be described below. In the embodiments of this application, the image recognition method provided in the embodiments of this application can be implemented by a server or a terminal as the execution subject, or it can be implemented through the interaction between the terminal and the server. The terminal is the terminal 110 in the above-described implementation environment, and the server is the server 140 in the above-described implementation environment. The interaction between the terminal and the server means that the terminal sends a sample dataset to the server, the server trains the image recognition model, and the server returns the trained image recognition model to the terminal. The embodiments of this application do not limit the execution subject. The following description will use a server as the execution subject as an example. See [link to documentation]. Figure 3 The methods include:

[0087] Figure 3 This is a flowchart of an image recognition method provided in an embodiment of this application. See also... Figure 3 The methods include:

[0088] 301. The server acquires multiple sub-images of the target 3D image.

[0089] The target 3D image is the 3D image to be recognized, and each sub-image of the target 3D image can be regarded as a local image of the target 3D image.

[0090] 302. Based on the multiple sub-images, the server obtains multiple temporal attention features and multiple spatial attention features of the target 3D image. The temporal attention features are used to represent the importance of the same position in different sub-images, and the spatial attention features are used to represent the importance of different positions in each sub-image.

[0091] The multiple temporal attention features of the target 3D image are determined based on the same locations in different sub-images, reflecting the importance of corresponding locations within different sub-images. The multiple spatial attention features of the target 3D image are determined based on different locations in each sub-image, reflecting the importance of different locations within each sub-image. During subsequent image recognition, predictions are focused on areas with higher importance, resulting in better prediction performance. Importance refers to the degree of contribution to the classification of the target 3D image; higher importance indicates a greater contribution to the classification, and lower importance indicates a lower contribution.

[0092] 303. The server fuses the multiple temporal attention features and the multiple spatial attention features to obtain the image features of the target's three-dimensional image.

[0093] The image features of the target 3D image are fused with multiple temporal attention features and multiple spatial attention features. This image feature can more accurately reflect the characteristics of the target 3D image, and the accuracy of determining the category of the target 3D image based on this image feature is also higher.

[0094] 304. The server determines the category of the target 3D image based on image features.

[0095] For 3D lung images, the category refers to the disease type, such as COVID-19, common pneumonia, or normal lungs. For 3D building images, the category refers to the building type. For 3D vehicle images, the category refers to the vehicle type.

[0096] The technical solution provided in this application divides a target 3D image into multiple sub-images during recognition, thereby achieving dimensionality reduction. Temporal and spatial attention features of the target 3D image are obtained based on these sub-images. Image features of the target 3D image are then extracted based on these temporal and spatial attention features. The feature extraction process does not require the participation of a 3D convolution kernel, resulting in image features with strong expressive power. Classifying the target 3D image based on these image features improves the efficiency of image recognition while ensuring accuracy.

[0097] It should be noted that steps 301-304 above are a brief introduction to the image recognition method provided in the embodiments of this application. The image recognition method provided in the embodiments of this application will be described in more detail below with some examples. See also Figure 4 The methods include:

[0098] Figure 4 This is a flowchart of an image recognition method provided in an embodiment of this application. See also... Figure 4 The methods include:

[0099] 401. The server acquires the 3D image of the target to be identified.

[0100] In one possible implementation, the server acquires the 3D image of the target to be identified uploaded by the terminal. Here, the server is server 140 in the aforementioned implementation environment, and the terminal is terminal 110 in the aforementioned implementation environment.

[0101] In this implementation, users can upload the target 3D image they want to recognize to the server via their terminal, and the server will then recognize the target 3D image. In other words, the server provides a cloud service for recognizing target 3D images, and terminals connected to the server via a network can use this cloud service. If the image recognition method provided in this application is applied to a scenario of recognizing 3D lung images, then the server also provides a cloud service for recognizing 3D lung images. After obtaining 3D lung images, hospitals can upload them to the server, which will then use the technical solution provided in this application to recognize the images and determine their category. This category can then help doctors in hospitals make diagnoses.

[0102] For example, a terminal acquires a 3D image of a target to be identified using a camera and sends an image recognition request to a server, the request carrying the 3D image of the target. The server receives the image recognition request and retrieves the 3D image of the target from it.

[0103] The above implementation methods will be described below in different scenarios.

[0104] Scenario 1: In a scenario involving the recognition of 3D lung images, the imaging device is a Computed Tomography (CT) scanner, and the 3D lung image is the target 3D image. The user (medical worker) can control the CT scanner to capture images of the patient's lungs via a terminal, which is also the control terminal for the CT scanner. After acquiring the 3D lung image, the terminal can upload it to the server, i.e., send an image recognition request to the server, carrying the 3D lung image. The server receives the image recognition request from the terminal and retrieves the 3D lung image from it.

[0105] Alternatively, a user (healthcare worker) can control a CT scanner via a terminal to take images of a patient's lungs, obtaining three-dimensional lung images. After acquiring multiple three-dimensional lung images, the user can select from them; the selected images constitute the target three-dimensional lung image. The user can then send an image recognition request, containing the desired three-dimensional lung image, to the server. The server receives the image recognition request and retrieves the desired three-dimensional lung image from it.

[0106] For example, see Figure 5The terminal runs an application for recognizing target 3D images, which provides an image recognition interface 501. The image recognition interface 501 displays an image selection control 502. In response to a click on the image selection control 502, the terminal displays an image selection area 503 on the image recognition interface, which displays multiple 3D lung images of the patient taken by a CT scanner. In response to selecting any 3D lung image, the terminal sends an image recognition request to the server, the request carrying the selected 3D lung image. The server receives the image recognition request sent by the terminal and retrieves the 3D lung image from it.

[0107] Alternatively, a user (healthcare worker) can control a CT scanner via a terminal to take images of the lungs of multiple patients, obtaining three-dimensional lung images of all patients. After acquiring the three-dimensional lung images of multiple patients, the terminal can send an image recognition request to the server, carrying the three-dimensional lung images of multiple patients in the request. The server receives the image recognition request sent by the terminal and extracts the three-dimensional lung images of multiple patients from the request.

[0108] Scenario 2: In a scenario involving the recognition of 3D vehicle images, the capturing device is a depth camera. After the terminal acquires multiple 3D vehicle images through the depth camera, the user can select from these images; the selected 3D vehicle image is the target 3D image. The user can send an image recognition request, carrying the 3D vehicle image, to the server via the terminal. The server receives the image recognition request and retrieves the 3D vehicle image from it.

[0109] It should be noted that in scenarios involving the recognition of 3D vehicle images, the terminal can also run an application for recognizing target 3D images, as described in Scenario 1. Users can select 3D vehicle images through this application.

[0110] Scenario 3: In a scenario involving the recognition of 3D material images, the imaging device is an atomic force microscope (AFM). For a batch of materials, the user can control the AFM via a terminal to capture images of the materials, obtaining multiple 3D material images. The user can select from these multiple images; the selected image is the target 3D image to be recognized. The user can send an image recognition request, containing the 3D material image, to the server via the terminal connected to the AFM. The server receives the image recognition request and retrieves the 3D material image from it.

[0111] It should be noted that in scenarios involving the recognition of 3D material images, the terminal can also run an application for recognizing target 3D images, as described in Scenario 1. Users can then select 3D material images using this application.

[0112] In one possible implementation, the server loads the 3D image of the target to be identified from a corresponding maintained database.

[0113] In this implementation, the server can store the target 3D image in a database, and when it is necessary to identify a target 3D image, it can be loaded directly from the database.

[0114] For example, a server maintains a database storing multiple target 3D images uploaded by users. The server loads the target 3D image to be identified from these multiple images. For instance, a terminal runs an application for target 3D image recognition, allowing users to upload target 3D images to the server's database. In some embodiments, the application can display multiple target 3D images stored in the database. When a target 3D image in the database needs to be identified, the user can select it using the application; the selected target 3D image is the one to be identified. In response to the selection of any target 3D image stored in the database, the terminal sends an image retrieval request to the server via the application. This request carries an identifier for the selected target 3D image. Upon receiving the image retrieval request, the server retrieves the identifier from the request, queries the database based on the identifier, and retrieves the target 3D image corresponding to that identifier; this target 3D image is the one to be identified.

[0115] In one possible implementation, the target 3D image is stored on a blockchain, and the server is able to retrieve the target 3D image from the blockchain.

[0116] In this implementation, due to the characteristics of blockchain, the target 3D image is difficult to tamper with after it is stored on the blockchain, thus ensuring the authenticity of the target 3D image.

[0117] 402. The server acquires multiple sub-images of the target 3D image.

[0118] In one possible implementation, the server divides the target 3D image into multiple slice images. Based on these multiple slice images, the server obtains multiple sub-images, each sub-image including at least one slice image.

[0119] To provide a clearer explanation of the above embodiments, the following description will be divided into two parts.

[0120] Part 1: The server divides the target's 3D image into multiple slice images.

[0121] In one possible implementation, the server performs image segmentation on the target 3D image to obtain multiple slice images of the target 3D image. In some embodiments, the slice images are 2D images. If the size of the target 3D image is H×W×C, where H is the height of the target 3D image, W is the width of the target 3D image, and C is the length of the target 3D image, and H, W, and C are all positive integers. For a slice image of the target 3D image, the size is H×W×1, and the target 3D image includes C slice images.

[0122] Part Two: The server obtains the multiple sub-images based on the multiple slice images.

[0123] In one possible implementation, the server groups every three adjacent slice images from the plurality of slice images into a sub-image, and every two adjacent sub-images include one or two identical slice images.

[0124] In the above implementation, the server groups every three adjacent slice images into a sub-image. This allows the three slice images in each sub-image to be treated as three channels of the sub-image during subsequent feature extraction, enabling direct feature extraction based on these three channels, resulting in high efficiency. Each pair of adjacent sub-images includes one or two identical slice images, ensuring information overlap between sub-images and preventing information loss during feature extraction.

[0125] For example, when a server divides a target 3D image into multiple slices, it can number these slices. Using the target 3D image as an example (size H×W×C) and the slices as H×W×1, the slices would be numbered 1-C. The server groups every three numbered slices into a sub-image. If every two adjacent sub-images contain the same slice—for example, if the server groups slices numbered 1, 2, and 3 into one sub-image, and slices numbered 3, 4, and 5 into another—then these two sub-images are adjacent and include slice number 3. If every two adjacent sub-images contain two identical slices—for example, if the server groups slices numbered 1, 2, and 3 into one sub-image, and slices numbered 2, 3, and 4 into another—then these two sub-images are also adjacent and include slices numbered 2 and 3. See, for example... Figure 6 , Figure 6 The system includes a target 3D image 601. The server slices the target 3D image 601 to obtain multiple slice images 602. The server combines every three adjacent slice images to obtain a sub-image 603.

[0126] 403. Based on the multiple sub-images, the server obtains multiple temporal attention features and multiple spatial attention features of the target 3D image. The temporal attention features are used to represent the importance of the same position in different sub-images, and the spatial attention features are used to represent the importance of different positions in each sub-image.

[0127] In one possible implementation, the server extracts features from each of the multiple sub-images to obtain a sub-image feature group for each sub-image. Based on the sub-image feature groups of the multiple sub-images, the server obtains multiple temporal attention features and multiple spatial attention features of the target 3D image.

[0128] To provide a clearer explanation of the above implementation methods, the following description will be divided into two parts.

[0129] The first part involves the server extracting features from each of the multiple sub-images to obtain sub-image feature groups for each sub-image.

[0130] In one possible implementation, for any one of the plurality of sub-images, the server divides the sub-image into multiple image blocks. If the sub-image includes three slice images, then an image block of the sub-image includes three regions at the same location of the three slices. The server performs feature extraction on each of the plurality of image blocks to obtain image block features for each image block. The image block features of the plurality of image blocks constitute the sub-image feature group of the sub-image. In some embodiments, the method of dividing the sub-image into image blocks and extracting features from the image blocks is also called PatchEmbedding.

[0131] For example, the server segments the pixel value matrix of the sub-image based on its size and the size of the image patch, obtaining multiple sub-pixel value matrices, each corresponding to an image patch. The size of the image patch is set by an engineer according to actual conditions; this embodiment does not limit this. A larger image patch size results in more image patches for the sub-image; a smaller image patch size results in fewer image patches. The server performs convolution processing on the multiple image patches of the sub-image to obtain image patch features for each patch. In some embodiments, the server uses vectors to represent the image patch features of each patch; that is, after convolving the sub-pixel value matrices of each patch, it obtains the image patch feature vectors for each patch.

[0132] For example, if a sub-image comprises three slice images, and the pixel value matrices of the three slice images are respectively as well as That is, the size of the sub-image is 4×4×3. If the size of the image patch is 2×2×3, then the server divides the pixel value matrix of the image, that is, it divides the pixel value matrix of the three slice images. as well as The pixels are divided into four sub-pixel value matrices. as well as These four sub-pixel value matrices are the sub-pixel value matrices corresponding to the four image patches of the sub-image. Each sub-pixel value matrix is ​​a three-dimensional matrix, consisting of three two-dimensional matrices. Each two-dimensional matrix can be considered a channel of the image patch. If the target three-dimensional image is a grayscale image, such as a CT image, then the value in each two-dimensional matrix is ​​the grayscale value; if the target three-dimensional image is a color image, then the value in each two-dimensional matrix is ​​the average of the three color channel values. The server inputs the sub-pixel value matrices of multiple image patches into the feature extraction model, and performs convolution processing on the sub-pixel value matrices of each image patch using the convolution kernel of the feature extraction model to obtain the image patch feature vector of each image patch. For a single image patch's sub-pixel value matrix, the server inputs it into the feature extraction model, and performs convolution processing on the three two-dimensional matrices of the sub-pixel matrix using the convolution kernel of the feature extraction model to obtain the feature vectors of the three two-dimensional matrices. The server fuses the feature vectors of the three two-dimensional matrices to obtain the image patch feature vector of the image patch. In some embodiments, the feature extraction model is a CNN (Convolutional Neural Networks). In some embodiments, the feature extraction model is a feature extraction model pre-trained on the ImageNet dataset.

[0133] In some embodiments, the server can use the following formula (1) to extract features from the image blocks of each sub-image to obtain the image feature group of each sub-image.

[0134]

[0135] Where z0 is the image feature group, Encoder is the feature extraction model, Z is the sub-pixel value matrix of the image patch, H is the height of the sub-pixel value matrix, W is the width of the sub-pixel value matrix, M is the number of slice images in each sub-image, and N is the number of image patches.

[0136] The second part involves the server acquiring multiple temporal attention features and multiple spatial attention features of the target 3D image based on the sub-image feature groups of the multiple sub-images.

[0137] In one possible implementation, the server obtains multiple temporal feature sequences from the sub-image feature groups of the multiple sub-images. These temporal feature sequences include image patch features corresponding to image blocks within each sub-image feature group. The server encodes these multiple temporal feature sequences using an attention mechanism to obtain temporal attention features for each temporal feature sequence. The server also encodes the image feature groups of the multiple sub-images using an attention mechanism to obtain spatial attention features for each image feature group.

[0138] To provide a clearer explanation of the above embodiments, the following description will be divided into several parts.

[0139] A. The server obtains multiple time feature sequences from the sub-image feature groups of the multiple sub-images.

[0140] If the target 3D image is regarded as a video, then multiple sub-images of the target 3D image can be regarded as multiple video frames of the target 3D image. Based on multiple video frames, multiple time feature sequences of the target 3D image can be obtained.

[0141] In one possible implementation, the server extracts image patch features of image blocks at the same location in each of the multiple sub-images from the sub-image feature groups. These image patch features from multiple image blocks at the same location form a temporal feature sequence. In other words, this temporal feature sequence is a set of image patch features from image blocks at the same location in the multiple sub-images. This temporal feature sequence corresponds to the image blocks at the same location in the multiple sub-images, and it can reflect the features of the same location at different depths in the target 3D image. Accordingly, multiple temporal feature sequences correspond to different locations in the target 3D image.

[0142] For example, the server extracts the image patch features of the top-left corner image patch of each of multiple sub-images from the sub-image feature groups. If there are three sub-images, the server extracts the image patch feature vectors of the top-left corner image patches of the three sub-images from the three sub-image feature groups, such as (1, 2, 3), (1, 3, 4), and (2, 3, 1). These three image patch feature vectors form a temporal feature sequence, corresponding to the top-left corner image patch of the sub-image. See also... Figure 7 , Figure 7 The image consists of three sub-images: image feature group 701, image feature group 702, and image feature group 703. The server combines the image patch features 7011 (top left corner of image feature group 701), 7021 (top left corner of image feature group 702), and 7031 (top left corner of image feature group 703) into a temporal feature sequence. Since the image patch features in the temporal feature sequence originate from the same location in different image feature groups, and multiple sub-images constitute the target 3D image, the temporal feature sequence can reflect the features of the same location in the target 3D image at different depths.

[0143] B. The server encodes the multiple time feature sequences based on an attention mechanism to obtain the time attention features of each time feature sequence.

[0144] In one possible implementation, the server obtains the key matrix, query matrix, and value matrix of each image patch feature in each temporal feature sequence. Based on the key matrix, query matrix, and value matrix of each image patch feature in each temporal feature sequence, the server obtains the temporal attention features of each temporal feature sequence.

[0145] For example, the server inputs one of multiple time feature sequences into an image recognition model. The model's temporal attention encoding layer processes this sequence, specifically through its three linear transformation matrices, to obtain the key matrix, query matrix, and value matrix of each image patch feature in the sequence. These three linear transformation matrices are the query transformation matrix WQ1, the key transformation matrix WK1, and the value transformation matrix WV1. The parameters of these transformation matrices are determined during the training of the image recognition model. In some embodiments, the combination of the three linear transformation matrices WQ1, WK1, and WV1 can also be referred to as the temporal attention encoding matrix U. qkv1 For a single image patch feature in the time feature sequence, the server performs dot product of the query matrix of the image patch feature with the key matrices of other image patches in the time feature sequence to obtain the temporal attention weights between the image patch feature and other image patch features. The server uses these temporal attention weights to multiply the value matrices of the corresponding image patch features to obtain the initial temporal attention features of the other image patch features on that image patch feature. The server fuses the initial temporal attention features of the other image patch features on that image patch feature to obtain the temporal attention features of the image patch feature. The server fuses the temporal attention features of multiple image patches to obtain the temporal attention features of the time feature sequence. The above process can be represented by formulas (2)-(4).

[0146]

[0147] Where x is the image patch feature, Q is the query matrix of the image patch feature, K is the key matrix of the image patch feature, V is the value matrix of the image patch feature, WQ1 is the query transformation matrix, WK1 is the key transformation matrix, and WV1 is the value transformation matrix.

[0148]

[0149] Where A is the temporal attention weight, softmax is the normalization function, and K... 1 Let D be the value matrix of other image patch features, where D is a constant.

[0150] S = A·V (4)

[0151] Where S represents the temporal attention feature of the image patch.

[0152] For example, given a temporal feature sequence (1,2)(1,3)(2,3), the three vectors represent the image patch features of the top-left corner image patches of the three sub-images. The server inputs the first image patch feature (1,2) from this temporal feature sequence into the image recognition model. The model processes this image patch feature (1,2) through its temporal attention coding layer, specifically through three linear transformation matrices, such as... as well as Right-multiplying the image patch feature (1, 2) yields the query matrix (1, -2), key matrix (2, -2), and value matrix (5, 4) for the image patch feature (1, 2). The server inputs the second image patch feature (1, 3) from this time feature sequence into the image recognition model. The model processes this image patch feature (1, 3) through its temporal attention encoding layer, specifically through three linear transformation matrices, such as... as well as Right-multiplying the image patch feature (1, 3) yields the query matrix (1, -3), key matrix (2, -3), and value matrix (6, 5) for the image patch feature (1, 3). The server inputs the third image patch feature (2, 3) from this time feature sequence into the image recognition model. The model processes this image patch feature (2, 3) through its temporal attention encoding layer, specifically through three linear transformation matrices, such as... as well as Multiplying the image patch feature (2, 3) by the right yields the query matrix (3, -3), key matrix (4, -3), and value matrix (9, 7) of the image patch feature (2, 3).

[0153] The server performs dot product on the query matrix (1, -2) of the first image patch feature (1, 2) with the key matrix (2, -3) of the second image patch feature (1, 3) and the key matrix (4, -3) of the third image patch feature (2, 3), respectively, to obtain (2, 6) and (4, -6). The server normalizes the two matrices (2, 6) and (4, -6) to obtain the temporal attention weights (0.25, 0.75) of the second image patch feature to the first image patch feature, and the temporal attention weights (0.4, -0.6) of the third image patch feature to the first image patch feature. In some embodiments, the server uses the Softmax function for normalization. The server performs dot product on the value matrix (6, 5) of the second image patch feature and the attention weights (0.25, 0.75) between the second and first image patch features to obtain the initial temporal attention feature (1.5, 3.75). The server performs a dot product between the value matrix (9, 7) of the third image patch feature and the attention weight (0.4, -0.6) between the third and first image patch features to obtain the initial temporal attention feature (3.6, -4.2). The server adds the value matrix (5, 4) of the first image patch feature to the two initial temporal attention features to obtain the temporal attention feature of the first image patch feature (10.1, 3.55).

[0154] The server performs dot product on the query matrix (1, -3) of the second image patch feature (1, 3) with the key matrix (2, -2) of the first image patch feature (1, 2) and the key matrix (4, -3) of the third image patch feature (2, 3), respectively, to obtain (8, 6) and (4, -5). The server normalizes the two matrices (8, 6) and (4, -5) to obtain the temporal attention weights (0.57, 0.43) of the first image patch feature on the second image patch feature and (0.44, -0.56) of the third image patch feature on the second image patch feature. The server performs dot product on the value matrix (5, 4) of the first image patch feature and the attention weights (0.57, 0.43) between the first and second image patch features to obtain the initial temporal attention feature (2.85, 1.72). The server performs a dot product between the value matrix (9, 7) of the third image patch feature and the attention weight (0.44, -0.56) between the third and second image patch features to obtain the initial temporal attention feature (4, -3.92). The server then adds the value matrix (6, 5) of the second image patch feature to the two initial temporal attention features to obtain the temporal attention feature of the second image patch feature (12.85, 2.8).

[0155] The server performs a dot product on the query matrix (3, -3) of the third image patch feature (2, 3) and the key matrices (2, -2) of the first image patch feature (1, 2) and the key matrices (2, -3) of the second image patch feature (1, 3), respectively, to obtain (6, 6) and (6, 9). The server normalizes the two matrices (6, 6) and (6, 9) to obtain the temporal attention weights (0.5, 0.5) of the first image patch feature on the third image patch feature and (0.4, 0.6) of the second image patch feature on the third image patch feature. The server performs a dot product on the value matrix (5, 4) of the first image patch feature and the attention weight (0.5, 0.5) between the first and third image patch features to obtain the initial temporal attention feature (2.5, 2). The server performs a dot product on the value matrix (6, 5) of the second image patch feature and the attention weight (0.4, 0.6) between the second and third image patch features to obtain the initial temporal attention feature (2.4, 3). The server adds the value matrix (9, 7) of the third image patch feature to the two initial temporal attention features to obtain the temporal attention feature (13, 9, 12) of the third image patch feature.

[0156] The server adds the temporal attention features of the three image patches together to obtain the temporal attention features (36.8, 18.35) of the temporal feature sequence.

[0157] It should be noted that in step B above, the example given is that the server performs attention encoding on the time feature sequence through a time attention encoding layer of the image recognition model. In other possible implementations, the image recognition model includes multiple time attention encoding layers, and the query transformation matrix, key transformation matrix, and value transformation matrix of different time attention encoding layers are different. The server can perform attention encoding on the time feature sequence through each time attention encoding layer to obtain multiple time attention features of the time feature sequence. The server performs a weighted summation of the multiple time attention features to obtain the final time attention feature of the time attention feature. The weights are set by the technician according to the actual situation, and this application embodiment does not limit this. The server can obtain the final time attention feature of the time attention feature through each time attention encoding layer using the following formula (5).

[0158]

[0159] Where MS represents the temporal attention feature of the time feature sequence, and A i V represents the temporal attention weight of the image patch feature numbered i in this temporal feature sequence. iLet U be the value matrix of the image patch feature numbered i in the time feature sequence, and U be the trainable parameter matrix.

[0160] In some embodiments, the process of a server using a single temporal attention coding layer to encode a temporal feature sequence can be referred to as single-head attention coding, while the process of a server using multiple temporal attention coding layers to encode a temporal feature sequence can be referred to as multi-head attention coding.

[0161] C. The server encodes the image feature groups of the multiple sub-images based on the attention mechanism to obtain the spatial attention features of each image feature group.

[0162] In one possible implementation, the server obtains the key matrix, query matrix, and value matrix of each image patch feature in each image feature group. Based on the key matrix, query matrix, and value matrix of each image patch feature in each image feature group, the server obtains the spatial attention features of each image feature group. See also Figure 7 , Figure 7 It includes image feature group 704, image feature group 705 and image feature group 706. Each image feature group includes multiple image block features. When encoding image feature group 704 to obtain the spatial attention features of image feature group 704, the features of each image block in image feature group 704 are used.

[0163] For example, the server inputs one of multiple image feature groups into an image recognition model. The spatial attention coding layer of the image recognition model processes this image feature group, specifically through its three linear transformation matrices, to obtain the key matrix, query matrix, and value matrix of each image patch feature in the feature group. The three linear transformation matrices are the query transformation matrix WQ2, the key transformation matrix WK2, and the value transformation matrix WV2. The parameters in these three linear transformation matrices are determined during the training of the image recognition model. In some embodiments, the combination of the three linear transformation matrices WQ2, WK2, and WV2 can also be referred to as the spatial attention coding matrix U. qkv2For a single image patch feature within the image feature group, the server performs dot product of the query matrix of this image patch feature with the key matrices of other image patches in the same group to obtain the spatial attention weights between this image patch feature and other image patch features. The server then uses these spatial attention weights to multiply the value matrices of the corresponding image patch features to obtain the initial spatial attention features of the other image patch features on this image patch feature. The server then fuses these initial spatial attention features to obtain the spatial attention features of the image patch feature. Finally, the server fuses the spatial attention features of multiple image patches to obtain the spatial attention features of the entire image feature group.

[0164] For example, for an image feature set (1,2)(2,3)(4,1), the three vectors represent the patch features of three image blocks in a sub-image. The server inputs the first patch feature (1,2) from this feature set into the recognition model. The spatial attention coding layer of the image recognition model processes this patch feature (1,2), that is, it processes it through the three linear transformation matrices of the spatial attention coding layer, for example, ... as well as Right-multiplying the image patch feature (1, 2) yields the query matrix (1, 2), key matrix (2, 2), and value matrix (4, 10) for the image patch feature (1, 2). The server inputs the second image patch feature (2, 3) from this image feature group into the image recognition model. The model processes this image patch feature (2, 3) through its spatial attention coding layer, specifically through three linear transformation matrices, such as... as well as Right-multiplying the image patch feature (2, 3) yields the query matrix (2, 3), key matrix (4, 3), and value matrix (7, 18) for the image patch feature (2, 3). The server inputs the third image patch feature (4, 1) from this image feature group into the image recognition model. The model processes this image patch feature (4, 1) through its spatial attention coding layer, specifically through three linear transformation matrices, such as... as well as Multiplying the image patch feature (4,1) by the right yields the query matrix (4,1), key matrix (8,1), and value matrix (9,12) of the image patch feature (4,1).

[0165] The server performs dot product on the query matrix (1,2) of the first image patch feature (1,2) with the key matrix (4,3) of the second image patch feature (2,3) and the key matrix (8,1) of the third image patch feature (4,1), respectively, to obtain (4,6) and (8,2). The server normalizes the two matrices (4,6) and (8,2) to obtain the spatial attention weights (0.4,0.6) of the second image patch feature to the first image patch feature and (0.8,0.2) of the third image patch feature to the first image patch feature. In some embodiments, the server uses the Softmax function for normalization. The server performs dot product on the value matrix (7,18) of the second image patch feature and the attention weights (0.4,0.6) between the second and first image patch features to obtain the initial spatial attention features (2.8,10.8). The server performs a dot product between the value matrix (9, 12) of the third image patch feature and the attention weight (0.8, 0.2) between the third and first image patch features to obtain the initial spatial attention feature (7.2, 2.4). The server adds the value matrix (4, 10) of the first image patch feature to the two initial spatial attention features to obtain the spatial attention feature (14, 23.2) of the first image patch feature.

[0166] The server performs dot product on the query matrix (2,3) of the second image patch feature (2,3) with the key matrix (2,2) of the first image patch feature (1,2) and the key matrix (8,1) of the third image patch feature (4,1), respectively, to obtain (4,6) and (16,3). The server normalizes the two matrices (4,6) and (16,3) to obtain the spatial attention weights (0.4, 0.6) of the first image patch feature on the second image patch feature and (0.84, 0.16) of the third image patch feature on the second image patch feature. The server performs dot product on the value matrix (4,10) of the first image patch feature and the attention weights (0.4, 0.6) between the first and second image patch features to obtain the initial spatial attention feature (1.6, 6). The server performs a dot product between the value matrix (9, 7) of the third image patch feature and the attention weight (0.84, 0.16) between the third and second image patch features to obtain the initial spatial attention feature (7.56, 1.12). The server then adds the value matrix (7, 18) of the second image patch feature to the two initial spatial attention features to obtain the spatial attention feature of the second image patch feature (16.16, 25.12).

[0167] The server performs dot product on the query matrix (4,1) of the third image patch feature (4,1) with the key matrix (2,2) of the first image patch feature (1,2) and the key matrix (4,3) of the second image patch feature (2,3), respectively, to obtain (8,2) and (16,3). The server normalizes the two matrices (8,2) and (16,3) to obtain the spatial attention weights (0.8, 0.2) of the first image patch feature on the third image patch feature and (0.84, 0.16) of the second image patch feature on the third image patch feature. The server performs dot product on the value matrix (4,10) of the first image patch feature and the attention weights (0.8, 0.2) between the first and third image patch features to obtain the initial spatial attention feature (3.2,2). The server performs a dot product between the value matrix (7, 18) of the second image patch feature and the attention weight (0.84, 0.16) between the second and third image patch features to obtain the initial spatial attention feature (5.88, 2.88). The server then adds the value matrix (9, 12) of the third image patch feature to the two initial spatial attention features to obtain the spatial attention feature of the third image patch feature (48.24, 16.88).

[0168] The server concatenates the spatial attention features of the three image patches to obtain the spatial attention features of the image feature group: (14, 23.2), (16.16, 25.12), and (48.24, 16.88).

[0169] It should be noted that step C above is illustrated by the example of the server performing attention encoding on an image feature group using a single spatial attention encoding layer of an image recognition model. In other possible implementations, the image recognition model includes multiple spatial attention encoding layers, with different query transformation matrices, key transformation matrices, and value transformation matrices. The server can perform attention encoding on the image feature group through each spatial attention encoding layer to obtain multiple spatial attention features of the image feature group. The server performs a weighted summation of the multiple spatial attention features to obtain the final spatial attention feature. The weights are set by the technician according to the actual situation, and this application embodiment does not limit this. In some embodiments, the process of the server using a single spatial attention encoding layer to perform attention encoding on an image feature group can be called single-head attention encoding, and the process of the server using multiple spatial attention encoding layers to perform attention encoding on an image feature group can be called multi-head attention encoding.

[0170] In one possible implementation, the server obtains multiple temporal feature sequences from the sub-image feature groups of the multiple sub-images. These temporal feature sequences include image patch features corresponding to image blocks in each sub-image feature group. The server encodes these multiple temporal feature sequences based on an attention mechanism to obtain temporal attention features for each temporal feature sequence. The server fuses the temporal attention features of each sub-image feature group and each temporal feature sequence to obtain a fused feature group for each sub-image. The server then encodes the fused feature group for each sub-image based on an attention mechanism to obtain spatial attention features for each sub-image.

[0171] The method by which the server obtains multiple temporal feature sequences from the sub-image feature groups of the multiple sub-images is described in step A of the previous embodiment and will not be repeated here. The server encodes the multiple temporal feature sequences based on an attention mechanism to obtain the temporal attention features of each temporal feature sequence. The method for this is described in step B of the previous embodiment and will not be repeated here. The following describes the method by which the server fuses the temporal attention features of each sub-image feature group and each temporal feature sequence to obtain the fused feature group of each sub-image.

[0172] In one possible implementation, the server adds the features of each image block in each sub-image feature group to the temporal attention features of the corresponding temporal feature sequence to obtain the fused features of each image block in each sub-image feature group. The fused features of multiple image block features in each sub-image feature group constitute the fused feature group of each sub-image.

[0173] The server encodes the fusion feature groups of each sub-image based on the attention mechanism to obtain the spatial attention features of each sub-image. The method is the same as step C in the previous implementation, and will not be repeated here.

[0174] In one possible implementation, the server uses a Transformer to encode multiple sub-images to obtain multiple temporal attention features and multiple spatial attention features of the target 3D image.

[0175] 404. The server fuses multiple temporal attention features and multiple spatial attention features to obtain the image features of the target 3D image.

[0176] In one possible implementation, the server concatenates multiple temporal attention features to obtain the concatenated temporal attention features of the target 3D image. The server fuses multiple spatial attention features to obtain the fused spatial attention features of the target 3D image. The server adds the concatenated temporal attention features and the fused spatial attention features of the target 3D image to obtain the spatiotemporal attention features of the target 3D image. The server performs fully connected processing on the spatiotemporal attention features of the target 3D image to obtain the image features of the target 3D image.

[0177] In this implementation, the acquisition of image features from the target 3D image combines temporal and spatial attention features, resulting in image features that more accurately reflect the characteristics of the target 3D image, which is helpful for subsequent classification. Furthermore, the acquisition of image features from the target 3D image does not employ 3D convolution, significantly reducing the computational load.

[0178] For example, the server concatenates multiple temporal attention features according to the positions of image patches corresponding to the temporal attention features in the target 3D image, obtaining the concatenated temporal attention features of the target 3D image. The server then performs a weighted sum of multiple spatial attention features to obtain the fused spatial attention features of the target 3D image. The weights are set by the technician according to the actual situation, and this embodiment does not limit this. The server adds the concatenated temporal attention features and the fused spatial attention features of the target 3D image to obtain the spatiotemporal attention features of the target 3D image. Finally, the server multiplies the spatiotemporal attention features of the target 3D image with a fully connected matrix to obtain the image features of the target 3D image.

[0179] For example, if one of the multiple temporal attention features corresponds to the image patch in the upper left corner of the target 3D image, the server sets that temporal attention feature in the upper left corner of the stitched temporal attention feature; if one of the multiple temporal attention features corresponds to the image patch in the lower right corner of the target 3D image, the server sets that temporal attention feature in the lower right corner of the stitched temporal attention feature. If the size of the target 3D image is H×W×C, then the size of the obtained stitched temporal attention feature is H×W. The server performs a weighted summation of multiple spatial attention features to obtain the fused spatial attention feature of the target 3D image; correspondingly, if the size of the target 3D image is H×W×C, then the size of the obtained fused spatial attention feature is H×W. The server adds the stitched temporal attention feature and the fused spatial attention feature of the target 3D image to obtain the spatiotemporal attention feature of the target 3D image. The server multiplies the spatiotemporal attention feature of the target 3D image with the fully connected matrix to obtain the image feature of the target 3D image.

[0180] 405. The server determines the category of the target 3D image based on image features.

[0181] In one possible implementation, the server performs fully connected processing and activation processing on the image features to obtain a probability distribution of the target 3D image. In response to any value in the probability distribution satisfying a target probability condition, the server determines the category corresponding to that value as the category of the target 3D image. In some embodiments, the server can use a Multi-Layer Perceptron (MLP) to perform fully connected processing and activation processing on the image features to obtain the probability distribution of the target 3D image.

[0182] For example, the server inputs image features into the image recognition layer 204 of the image recognition model, multiplies the classification matrix of the image recognition layer 204 with the image features, and obtains the probability distribution of the target 3D image. In response to any value in the probability distribution being greater than or equal to a probability threshold, or any value being the largest among multiple values ​​in the probability distribution, the category corresponding to that value is determined as the category of the target 3D image.

[0183] For example, a classifier can determine the category of a target 3D image based on image features using the following formula (6).

[0184] p = w T R (6)

[0185] Where p is the probability distribution of the target 3D image, ω T This is the classification matrix of the classifier.

[0186] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0187] During the experiment, the image recognition method provided in this application was applied to the scenario of lung image recognition, and it was able to quickly and accurately identify COVID-19, common pneumonia, and normal cases. The image recognition method provided in this application can be used in different CT devices for assisted diagnosis.

[0188] The dataset used in the experiment was COVID-CT-MD (COVID-19 CT dataset). This dataset contains a total of 307 CT images, of which 171 are confirmed cases of COVID-19, 76 are normal, and the remaining 60 are cases of ordinary pneumonia. Each CT image is approximately 512*512*128 pixels in size.

[0189] The experimental results are shown in Table 1. The image recognition method provided in this application embodiment can achieve an accuracy of 91.80%, which meets the accuracy requirements of CT image-assisted diagnosis.

[0190] Table 1

[0191]

[0192] The technical solution provided in this application divides a target 3D image into multiple sub-images during recognition, thereby achieving dimensionality reduction. Temporal and spatial attention features of the target 3D image are obtained based on these sub-images. Image features of the target 3D image are then extracted based on these temporal and spatial attention features. The feature extraction process does not require the participation of a 3D convolution kernel, resulting in image features with strong expressive power. Classifying the target 3D image based on these image features improves the efficiency of image recognition while ensuring accuracy.

[0193] In addition to the image recognition method described in steps 401-405 above, this application embodiment also provides another image recognition method for training an image recognition model. See [link to relevant documentation]. Figure 8 The methods include:

[0194] 801. The server acquires multiple sample sub-images of the sample 3D image.

[0195] Each sample 3D image corresponds to a labeled category. The goal of training the image recognition model is to ensure that the predicted category output by the image recognition model is the same as the labeled category of the sample 3D image after the sample 3D image is input into the trained image recognition model.

[0196] Furthermore, the method by which the server acquires multiple sample sub-images of the sample 3D image is based on the same inventive concept as the method by which the server acquires multiple sub-images of the target 3D image in step 402 above. The implementation process is described in the relevant description of step 402 above, and will not be repeated here.

[0197] 802. The server inputs multiple sample sub-images into the image recognition model. Based on the multiple sample sub-images, the image recognition model obtains multiple temporal attention features and multiple spatial attention features of the sample 3D image. The temporal attention features are used to represent the importance of the same position in different sub-images, and the spatial attention features are used to represent the importance of different positions in each sub-image.

[0198] The process of the server acquiring multiple temporal attention features and multiple spatial attention features of the sample 3D image belongs to the same inventive concept as the process of acquiring multiple temporal attention features and multiple spatial attention features of the target 3D image in step 403 above. The implementation process is described in the relevant description of step 403 above, and will not be repeated here.

[0199] 803. The server fuses multiple temporal attention features and multiple spatial attention features through an image recognition model to obtain the image features of the sample 3D image.

[0200] The process by which the server fuses multiple temporal attention features and multiple spatial attention features through an image recognition model to obtain the image features of the sample 3D image is the same inventive concept as the process in step 404 above where the server fuses multiple temporal attention features and multiple spatial attention features to obtain the image features of the target 3D image. The implementation process is described in the relevant description of step 404 above and will not be repeated here.

[0201] 804. The server uses an image recognition model to output the predicted category of the sample 3D image based on image features.

[0202] The process by which the server outputs the predicted category of the sample 3D image based on image features through an image recognition model is the same inventive concept as the process of determining the category of the target 3D image based on image features in step 405 above. The implementation process is described in the relevant description of step 405 above, and will not be repeated here.

[0203] 805. The server adjusts the model parameters of the image recognition model based on the difference information between the predicted category and the labeled category of the sample 3D image.

[0204] The method by which the server outputs the first predicted category of the sample three-dimensional image based on the features of the sample three-dimensional image through the image recognition model belongs to the same inventive concept as step 406 above. The implementation process is described in the relevant description of step 406 above, and will not be repeated here.

[0205] It should be noted that, in the process of adjusting the model parameters of the image recognition model based on the difference information, the server can use the gradient descent method, such as batch gradient descent, stochastic gradient descent, Adam, or mini-batch gradient descent. This application embodiment does not limit this method.

[0206] In the experiment, the original data was divided into training and testing sets according to an 80:20 ratio. The Adam optimizer was used during training, with a batch size of 3, a learning rate of 0.0001, and a training duration of 100 epochs.

[0207] The technical solution provided in this application divides a sample 3D image into multiple sub-images during image recognition model training to achieve dimensionality reduction. Temporal and spatial attention features of the sample 3D image are obtained based on these sub-images. Image features of the sample 3D image are then extracted based on these temporal and spatial attention features. The feature extraction process does not require the participation of a 3D convolution kernel, resulting in image features with strong expressive power. Class prediction of the sample 3D image is performed based on these image features. The image recognition model is trained based on the difference between the predicted and labeled classes. Using the trained image recognition model, the efficiency of image recognition can be improved while ensuring accuracy.

[0208] Figure 9 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application. See also... Figure 9 The device includes: a first sub-image acquisition module 901, a first feature acquisition module 902, a first feature fusion module 903, and a first category determination module 904.

[0209] The first sub-image acquisition module 901 is used to acquire multiple sub-images of the target 3D image;

[0210] The first feature acquisition module 902 is used to acquire multiple temporal attention features and multiple spatial attention features of the target 3D image based on the multiple sub-images. The temporal attention features are used to represent the importance of the same position in different sub-images, and the spatial attention features are used to represent the importance of different positions in each sub-image.

[0211] The first feature fusion module 903 is used to fuse the multiple temporal attention features and the multiple spatial attention features to obtain the image features of the target three-dimensional image;

[0212] The first category determination module 904 is used to determine the category of the target three-dimensional image based on the image features.

[0213] In one possible implementation, the first sub-image acquisition module 901 is used to divide the target three-dimensional image into multiple slice images; and acquire the multiple sub-images based on the multiple slice images, wherein each sub-image includes at least one of the slice images.

[0214] In one possible implementation, the first sub-image acquisition module 901 is used to form a sub-image from every three adjacent slice images in the plurality of slice images, wherein every two adjacent sub-images in the plurality of sub-images include one or two identical slice images.

[0215] In one possible implementation, the first feature acquisition module 902 is used to extract features from the plurality of sub-images respectively to obtain sub-image feature groups for each of the sub-images; and based on the sub-image feature groups of the plurality of sub-images, to acquire a plurality of temporal attention features and a plurality of spatial attention features of the target three-dimensional image.

[0216] In one possible implementation, the first feature acquisition module 902 is used to divide any one of the plurality of sub-images into a plurality of image blocks; extract features from each of the plurality of image blocks to obtain image block features of each image block, and the image block features of the plurality of image blocks form a sub-image feature group of the sub-image.

[0217] In one possible implementation, the first feature acquisition module 902 is configured to acquire multiple temporal feature sequences from the sub-image feature groups of the multiple sub-images, wherein the temporal feature sequences include image block features corresponding to image blocks in each of the sub-image feature groups; encode the multiple temporal feature sequences based on an attention mechanism to obtain temporal attention features of each of the temporal feature sequences; and encode the image feature groups of the multiple sub-images based on an attention mechanism to obtain spatial attention features of each of the image feature groups.

[0218] In one possible implementation, the first feature acquisition module 902 is used to acquire the key matrix, query matrix, and value matrix of each image block feature in each of the time feature sequences; and to acquire the time attention features of each of the time feature sequences based on the key matrix, query matrix, and value matrix of each image block feature in each of the time feature sequences.

[0219] In one possible implementation, the first feature acquisition module 902 is configured to acquire multiple temporal feature sequences from the sub-image feature groups of the multiple sub-images, wherein the temporal feature sequences include image block features corresponding to image blocks in each of the sub-image feature groups; encode the multiple temporal feature sequences based on an attention mechanism to obtain temporal attention features of each of the temporal feature sequences; fuse each of the sub-image feature groups and the temporal attention features of each of the temporal feature sequences to obtain a fused feature group of each of the sub-images; and encode the fused feature group of each of the sub-images based on an attention mechanism to obtain spatial attention features of each of the sub-images.

[0220] In one possible implementation, the first feature acquisition module 902 is used to acquire the key matrix, query matrix, and value matrix of each image block feature in each of the image feature groups; and to acquire the spatial attention features of each of the image feature groups based on the key matrix, query matrix, and value matrix of each image block feature in each of the image feature groups.

[0221] In one possible implementation, the first category determination module 904 is used to perform fully connected processing and activation processing on the image features to obtain a probability distribution column of the target 3D image; in response to any value in the probability distribution column meeting the target probability condition, the category corresponding to the value is determined as the category of the target 3D image.

[0222] In one possible implementation, the first feature acquisition module 902 is used to encode the plurality of sub-images using a Transformer to obtain a plurality of temporal attention features and a plurality of spatial attention features of the target 3D image.

[0223] It should be noted that the image recognition device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image recognition device and the image recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0224] The technical solution provided in this application divides a target 3D image into multiple sub-images during recognition, thereby achieving dimensionality reduction. Temporal and spatial attention features of the target 3D image are obtained based on these sub-images. Image features of the target 3D image are then extracted based on these temporal and spatial attention features. The feature extraction process does not require the participation of a 3D convolution kernel, resulting in image features with strong expressive power. Classifying the target 3D image based on these image features improves the efficiency of image recognition while ensuring accuracy.

[0225] Figure 10 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application. See also... Figure 10 The device includes: a second sub-image acquisition module 1001, a second category determination module 1002, and an adjustment module 1003.

[0226] The second sub-image acquisition module 1001 is used to acquire multiple sample sub-images of the sample 3D image;

[0227] The second category determination module 1002 is used to input the multiple sample sub-images into an image recognition model, and through the image recognition model, based on the multiple sample sub-images, obtain multiple temporal attention features and multiple spatial attention features of the sample 3D image. The temporal attention features are used to represent the importance of the same position in different sub-images, and the spatial attention features are used to represent the importance of different positions in each sub-image. The multiple temporal attention features and the multiple spatial attention features are fused to obtain the image features of the sample 3D image. Based on the image features, the predicted category of the sample 3D image is output.

[0228] The adjustment module 1003 is used to adjust the model parameters of the image recognition model based on the difference information between the predicted category and the labeled category of the sample 3D image.

[0229] It should be noted that the image recognition device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image recognition device and the image recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0230] The technical solution provided in this application divides a sample 3D image into multiple sub-images during image recognition model training to achieve dimensionality reduction. Temporal and spatial attention features of the sample 3D image are obtained based on these sub-images. Image features of the sample 3D image are then extracted based on these temporal and spatial attention features. The feature extraction process does not require the participation of a 3D convolution kernel, resulting in image features with strong expressive power. Class prediction of the sample 3D image is performed based on these image features. The image recognition model is trained based on the difference between the predicted and labeled classes. Using the trained image recognition model, the efficiency of image recognition can be improved while ensuring accuracy.

[0231] This application provides a computer device for performing the above-described method. This computer device can be implemented as a terminal or a server. The structure of the terminal will be described below:

[0232] Figure 11 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal 1100 can be a smartphone, tablet computer, laptop computer, desktop computer, smartwatch, etc., but is not limited to these.

[0233] Typically, terminal 1100 includes one or more processors 1111 and one or more memories 1102.

[0234] Processor 1111 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1111 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1111 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1111 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1111 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0235] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one computer program, which is executed by the processor 1111 to implement the image recognition method provided in the method embodiments of this application.

[0236] In some embodiments, the terminal 1100 may also optionally include: a peripheral device interface 1103 and at least one peripheral device. The processor 1111, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, a positioning assembly 1108, and a power supply 1109.

[0237] Peripheral device interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1111 and memory 1102. In some embodiments, processor 1111, memory 1102 and peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1111, memory 1102 and peripheral device interface 1103 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0238] The radio frequency (RF) circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc.

[0239] Display screen 1105 is used to display a user interface (UI). This UI may include graphics, text, icons, video, and any combination thereof. When display screen 1105 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1111 for processing. In this case, display screen 1105 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard.

[0240] The camera assembly 1106 is used to capture images or videos. Optionally, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal.

[0241] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 1111 for processing, or input to the radio frequency circuit 1104 to realize voice communication.

[0242] The positioning component 1108 is used to locate the current geographical location of the terminal 1100 in order to enable navigation or LBS (Location Based Service).

[0243] The power supply 1109 is used to supply power to the various components in the terminal 1100. The power supply 1109 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery.

[0244] In some embodiments, the terminal 1100 further includes one or more sensors 1110. The one or more sensors 1110 include, but are not limited to: an accelerometer 1111, a gyroscope 1112, a pressure sensor 1113, a fingerprint sensor 1114, an optical sensor 1115, and a proximity sensor 1116.

[0245] Accelerometer 1111 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established with terminal 1100.

[0246] The gyroscope sensor 1112 can detect the orientation and rotation angle of the terminal 1100. The gyroscope sensor 1112 can work in conjunction with the accelerometer sensor 1111 to collect the user's 3D movements on the terminal 1100.

[0247] The pressure sensor 1113 can be installed on the side bezel of the terminal 1100 and / or on the lower layer of the display screen 1105. When the pressure sensor 1113 is installed on the side bezel of the terminal 1100, it can detect the user's grip signal on the terminal 1100, and the processor 1111 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1113. When the pressure sensor 1113 is installed on the lower layer of the display screen 1105, the processor 1111 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1105.

[0248] The fingerprint sensor 1114 is used to collect the user's fingerprint. The processor 1111 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1114, or the fingerprint sensor 1114 identifies the user's identity based on the collected fingerprint.

[0249] An optical sensor 1115 is used to collect ambient light intensity. In one embodiment, a processor 1111 can control the display brightness of a display screen 1105 based on the ambient light intensity collected by the optical sensor 1115.

[0250] The proximity sensor 1116 is used to detect the distance between the user and the front of the terminal 1100.

[0251] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on terminal 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0252] The aforementioned computer equipment can also be implemented as a server. The structure of a server is described below:

[0253] Figure 12 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1200 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1201 and one or more memories 1202. The one or more memories 1202 store at least one computer program, which is loaded and executed by the one or more processors 1201 to implement the methods provided in the above-described method embodiments. Of course, the server 1200 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1200 may also include other components for implementing device functions, which will not be elaborated upon here.

[0254] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the image recognition method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0255] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the image recognition method described above.

[0256] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0257] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0258] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image recognition method, characterized in that, The method includes: Obtain multiple sub-images of the target 3D image; Feature extraction is performed on each of the multiple sub-images to obtain sub-image feature groups for each sub-image. Multiple temporal feature sequences are obtained from these sub-image feature groups, each temporal feature sequence including image block features corresponding to image patches within each sub-image feature group. The multiple temporal feature sequences are then encoded using an attention mechanism to obtain temporal attention features for each temporal feature sequence. Finally, the sub-image feature groups of the multiple sub-images are encoded using an attention mechanism to obtain spatial attention features for each sub-image feature group. The temporal attention features represent the importance of the same position in different sub-images, and the spatial attention features represent the importance of different positions in different sub-images. Multiple temporal attention features and multiple spatial attention features are fused to obtain the image features of the target 3D image; Based on the image features, the category of the target 3D image is determined.

2. The method according to claim 1, characterized in that, The multiple sub-images of the target 3D image include: The target 3D image is divided into multiple slice images; Based on the plurality of slice images, the plurality of sub-images are obtained, each of the sub-images including at least one of the slice images.

3. The method according to claim 2, characterized in that, The process of obtaining the multiple sub-images based on the multiple slice images includes: In the plurality of slice images, every three adjacent slice images form a sub-image, and in the plurality of sub-images, every two adjacent sub-images include one or two identical slice images.

4. The method according to claim 1, characterized in that, The step of extracting features from each of the multiple sub-images to obtain sub-image feature groups for each sub-image includes: For any one of the plurality of sub-images, the sub-image is divided into a plurality of image blocks; Feature extraction is performed on each of the multiple image blocks to obtain image block features for each image block. The image block features of the multiple image blocks form the sub-image feature group of the sub-image.

5. The method according to claim 1, characterized in that, The encoding of the multiple time feature sequences based on the attention mechanism to obtain the time attention features of each time feature sequence includes: Obtain the key matrix, query matrix, and value matrix of each image block feature in each of the aforementioned time feature sequences; Based on the key matrix, query matrix, and value matrix of each image block feature in each of the time feature sequences, the temporal attention features of each of the time feature sequences are obtained.

6. The method according to claim 1, characterized in that, The method further includes: Multiple temporal feature sequences are obtained from the sub-image feature groups of the multiple sub-images, and the temporal feature sequences include the image block features of the corresponding image blocks in each of the sub-image feature groups; The multiple time feature sequences are encoded based on an attention mechanism to obtain the time attention features of each time feature sequence; The feature groups of each sub-image and the temporal attention features of each temporal feature sequence are fused to obtain the fused feature groups of each sub-image. The fusion feature groups of each sub-image are encoded based on the attention mechanism to obtain the spatial attention features of each sub-image.

7. The method according to claim 1, characterized in that, Determining the category of the target 3D image based on the image features includes: The image features are subjected to fully connected processing and activation processing to obtain the probability distribution column of the target 3D image; In response to any value in the probability distribution column meeting the target probability condition, the category corresponding to the value is determined as the category of the target 3D image.

8. The method according to claim 1, characterized in that, The method further includes: The multiple sub-images are encoded using a Transformer to obtain multiple temporal attention features and multiple spatial attention features of the target 3D image.

9. An image recognition method, characterized in that, The method includes: Obtain multiple sample sub-images of the sample 3D image; The multiple sample sub-images are input into an image recognition model. The model extracts features from each sample sub-image to obtain sub-image feature groups for each sub-image. Multiple temporal feature sequences are obtained from these sub-image feature groups, each temporal feature sequence including image patch features corresponding to image blocks within each sub-image feature group. These temporal feature sequences are then encoded using an attention mechanism to obtain temporal attention features for each temporal feature sequence. Similarly, the sub-image feature groups are encoded using an attention mechanism to obtain spatial attention features for each sub-image feature group. The temporal attention features represent the importance of the same position in different sub-images, while the spatial attention features represent the importance of different positions in different sub-images. The multiple temporal and spatial attention features are then fused to obtain the image features of the sample 3D image. Based on these image features, the predicted category of the sample 3D image is output. The model parameters of the image recognition model are adjusted based on the difference information between the predicted category and the labeled category of the sample 3D image.

10. An image recognition device, characterized in that, The device includes: The first sub-image acquisition module is used to acquire multiple sub-images of the target 3D image; The first feature acquisition module is used to extract features from the plurality of sub-images respectively to obtain sub-image feature groups for each sub-image; to obtain a plurality of temporal feature sequences from the sub-image feature groups of the plurality of sub-images, wherein the temporal feature sequences include image block features corresponding to image blocks in each sub-image feature group; to encode the plurality of temporal feature sequences based on an attention mechanism to obtain temporal attention features for each temporal feature sequence; and to encode the sub-image feature groups of the plurality of sub-images based on an attention mechanism to obtain spatial attention features for each sub-image feature group, wherein the temporal attention features are used to represent the importance of the same position in different sub-images, and the spatial attention features are used to represent the importance of different positions in each sub-image. The first feature fusion module is used to fuse multiple temporal attention features and multiple spatial attention features to obtain the image features of the target three-dimensional image; The first category determination module is used to determine the category of the target 3D image based on the image features.

11. The apparatus according to claim 10, characterized in that, The first sub-image acquisition module is used to divide the target three-dimensional image into multiple slice images; and to acquire multiple sub-images based on the multiple slice images, wherein each sub-image includes at least one of the slice images.

12. The apparatus according to claim 11, characterized in that, The first sub-image acquisition module is used to form a sub-image from every three adjacent slice images in the plurality of slice images, wherein every two adjacent sub-images in the plurality of sub-images include one or two identical slice images.

13. The apparatus according to claim 10, characterized in that, The first feature acquisition module is used to divide any one of the plurality of sub-images into a plurality of image blocks; to extract features from each of the plurality of image blocks to obtain image block features of each image block, and the image block features of the plurality of image blocks form a sub-image feature group of the sub-image.

14. The apparatus according to claim 10, characterized in that, The first feature acquisition module is used to acquire the key matrix, query matrix, and value matrix of each image block feature in each of the time feature sequences; and to acquire the time attention features of each of the time feature sequences based on the key matrix, query matrix, and value matrix of each image block feature in each of the time feature sequences.

15. The apparatus according to claim 10, characterized in that, The first feature acquisition module is used to acquire multiple time feature sequences from the sub-image feature groups of the multiple sub-images, wherein the time feature sequences include image block features of corresponding image blocks in each of the sub-image feature groups; The multiple time feature sequences are encoded based on an attention mechanism to obtain the time attention features of each time feature sequence; the feature groups of each sub-image and the time attention features of each time feature sequence are fused to obtain the fused feature groups of each sub-image. The fusion feature groups of each sub-image are encoded based on the attention mechanism to obtain the spatial attention features of each sub-image.

16. The apparatus according to claim 10, characterized in that, The first category determination module is used to perform fully connected processing and activation processing on the image features to obtain a probability distribution column of the target 3D image; in response to any value in the probability distribution column meeting the target probability condition, the category corresponding to the value is determined as the category of the target 3D image.

17. The apparatus according to claim 10, characterized in that, The first feature acquisition module is used to encode the multiple sub-images using a Transformer to obtain multiple temporal attention features and multiple spatial attention features of the target 3D image.

18. An image recognition device, characterized in that, The device includes: The second sub-image acquisition module is used to acquire multiple sample sub-images of the sample 3D image; The second category determination module is used to input the multiple sample sub-images into an image recognition model, and through the image recognition model, extract features from the multiple sample sub-images to obtain sub-image feature groups for each sample sub-image. Multiple temporal feature sequences are obtained from the sub-image feature groups, each temporal feature sequence including image block features corresponding to image blocks in each sub-image feature group. The multiple temporal feature sequences are encoded based on an attention mechanism to obtain temporal attention features for each temporal feature sequence. The sub-image feature groups of the multiple sample sub-images are also encoded based on an attention mechanism to obtain spatial attention features for each sub-image feature group. The temporal attention features represent the importance of the same position in different sub-images, and the spatial attention features represent the importance of different positions in each sub-image. The multiple temporal attention features and multiple spatial attention features are fused to obtain the image features of the sample 3D image. Based on the image features, the predicted category of the sample 3D image is output. The adjustment module is used to adjust the model parameters of the image recognition model based on the difference information between the predicted category and the labeled category of the sample 3D image.

19. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the image recognition method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the image recognition method as described in any one of claims 1 to 9.

Images