A processing method and device based on a neural network model, equipment and medium

Abstract

Embodiments of the present application provide a processing method and device based on a neural network model, equipment and medium, the method comprises: obtaining sample image data, and training a neural network model using the sample image data; obtaining target image data, and predicting the target image data using the trained neural network model to obtain a prediction result of the target image data, the trained neural network model is used for feature extraction of the target image data in multiple stages to obtain an image depth feature vector, performing a global average pooling operation on the obtained image depth feature vector, and obtaining a prediction class probability of the target image data through a classifier after passing through a fully connected layer, selecting the class with the maximum prediction class probability as the prediction result of the target image data, which realizes optimization of the neural network model used for image analysis, reduces the requirement of the neural network model for hardware computing power, and improves the accuracy of image analysis.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a processing method, apparatus, device, and medium based on a neural network model. Background Technology

[0002] With the development of science and technology, neural network models are gradually being applied in various scenarios, especially in image analysis, where the application of neural network models improves the accuracy of image analysis.

[0003] However, existing neural network models used for image analysis typically require high hardware computing power and lack sufficient analysis accuracy, resulting in poor performance in some scenarios. Summary of the Invention

[0004] In view of the above problems, a processing method, apparatus, device, and medium based on a neural network model are proposed to overcome or at least partially solve the above problems, including:

[0005] A processing method based on a neural network model, the method comprising:

[0006] Acquire sample image data and use the sample image data to train a neural network model;

[0007] The target image data is acquired, and a trained neural network model is used to predict the target image data to obtain the prediction result of the target image data. The trained neural network model is used to extract features from the target image data in multiple stages to obtain image depth feature vectors, perform global average pooling on the obtained image depth feature vectors, and obtain the predicted class probability of the target image data through a classifier after passing through a fully connected layer. The class with the highest predicted class probability is selected as the prediction result of the target image data.

[0008] Optionally, the plurality of stages includes a first stage, the feature extraction process of the first stage including:

[0009] The target image data is processed by ordinary convolution, and the resulting feature vector is processed by a first lightweight convolution block to obtain the feature vector output by the first stage.

[0010] The first lightweight convolutional block is processed as follows:

[0011] The input feature vector is increased in dimensionality to obtain a high-dimensional feature vector. The high-dimensional feature vector is then subjected to a depthwise convolution operation to obtain a feature map. The feature map is then subjected to an average pooling operation to obtain a feature vector with a certain number of channels. The feature vector is then passed through a fully connected layer with an activation function to obtain matrix attention vector weights. The matrix attention vector weights are then multiplied element-wise with the feature map to obtain a new feature map, which is then reduced in dimensionality. Finally, the reduced-dimensional feature vector is residually concatenated with the input feature vector to obtain the output feature vector.

[0012] Optionally, the plurality of stages includes a second stage, wherein the feature extraction process of the second stage includes:

[0013] The feature vector output from the first stage is mapped and input into the second stage, processed by the second lightweight convolutional block, and the resulting feature vector is processed by the first lightweight convolutional block to obtain the feature vector output from the second stage.

[0014] The second lightweight convolutional block is processed as follows:

[0015] The input feature map is increased in dimensionality by increasing the number of channels, and then the spatial size is reduced by depthwise convolution to obtain the feature map. The obtained feature map is then subjected to global pooling to obtain the feature vector of the number of channels. The obtained feature vector is then passed through a fully connected layer with an activation function to obtain the matrix attention vector weights. The obtained matrix attention vector weights are then multiplied element-wise with the feature map to obtain the feature map. Finally, dimensionality reduction is performed to obtain the feature vector with reduced spatial size.

[0016] Optionally, the plurality of stages includes a third stage, the feature extraction process of the third stage including:

[0017] The second lightweight convolutional block is used to downsample the feature vector output by the second stage in terms of spatial size. The sampled feature vector is then input into the local-global fusion module to fuse the global and local features of the feature vector to obtain a fused feature vector. The fused feature vector is then residually connected with the feature vector output by the second stage to obtain the feature vector output by the third stage.

[0018] The local-to-global fusion module employs the following processing:

[0019] The input feature map is amplified in terms of channel number, and then the feature vector is input into the local feature extraction block and the global feature extraction block respectively to obtain the local feature vector and the global feature vector. The feature is then fused by convolution of size 1x1 to obtain the fused feature vector.

[0020] Optionally, the plurality of stages includes a fourth stage, the feature extraction process of the fourth stage including:

[0021] The features output from the third stage are reduced in spatial dimension by a second lightweight convolutional block, and the resulting feature vector is then input into a local feature fusion block to obtain a local-global fused feature vector, which is used as the feature vector output from the fourth stage.

[0022] Optionally, the plurality of stages includes a fifth stage, the feature extraction process of the fifth stage including:

[0023] The feature vector output from the fourth stage is input into the second lightweight convolutional block for spatial downsampling, and the resulting feature vector is input into the local-global fusion block to obtain the fused feature vector. The resulting fused feature vector is then passed through a convolutional layer with a convolutional size of 1x1 to obtain the image depth feature vector.

[0024] Optionally, before training the neural network model using the sample image data, the method further includes:

[0025] The sample image data is preprocessed.

[0026] A processing device based on a neural network model, the device being used for:

[0027] Acquire sample image data and use the sample image data to train a neural network model;

[0028] The target image data is acquired, and a trained neural network model is used to predict the target image data to obtain the prediction result of the target image data. The trained neural network model is used to extract features from the target image data in multiple stages to obtain image depth feature vectors, perform global average pooling on the obtained image depth feature vectors, and obtain the predicted class probability of the target image data through a classifier after passing through a fully connected layer. The class with the highest predicted class probability is selected as the prediction result of the target image data.

[0029] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the processing method based on a neural network model as described above.

[0030] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the processing method based on a neural network model as described above.

[0031] The embodiments of the present invention have the following advantages:

[0032] In this embodiment of the invention, sample image data is acquired and used to train a neural network model. Target image data is then acquired, and the trained neural network model is used to predict the target image data, resulting in a prediction result. The trained neural network model performs multi-stage feature extraction on the target image data to obtain an image depth feature vector. Global average pooling is applied to the obtained image depth feature vector, and after passing through a fully connected layer, a classifier is used to obtain the predicted class probability of the target image data. The class with the highest predicted class probability is selected as the prediction result of the target image data. This optimizes the neural network model used for image analysis, reduces the hardware computing power requirements of the neural network model, and improves the accuracy of image analysis. Attached Figure Description

[0033] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of the steps of a processing method based on a neural network model provided in an embodiment of the present invention;

[0035] Figure 2a This is a schematic diagram illustrating the details of a network model architecture provided in an embodiment of the present invention;

[0036] Figure 2b This is a schematic diagram of a model flow provided in an embodiment of the present invention;

[0037] Figure 2c This is a schematic diagram of a first lightweight convolutional block provided in an embodiment of the present invention;

[0038] Figure 2d This is a schematic diagram of a second lightweight convolutional block provided in an embodiment of the present invention;

[0039] Figure 2e This is a schematic diagram of a local-to-global fusion module provided in an embodiment of the present invention;

[0040] Figure 2f This is a schematic diagram of a global feature extraction block provided in an embodiment of the present invention;

[0041] Figure 2g This is a schematic diagram of a local feature extraction block provided in an embodiment of the present invention;

[0042] Figure 3This is a flowchart of another processing method based on a neural network model provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0044] Reference Figure 1 The diagram illustrates a flowchart of a processing method based on a neural network model according to an embodiment of the present invention, which may specifically include the following steps:

[0045] Step 101: Obtain sample image data and use the sample image data to train a neural network model.

[0046] In some embodiments, the image data (including sample image data and target image data hereinafter) can be any image data, such as vehicle images, face images, etc.

[0047] In some embodiments, in the context of medical image processing, the image data can be medical image data, such as brain PET (Positron Emission Tomography), which is an imaging technique that reflects the genetic, molecular, metabolic, and functional states of brain lesions.

[0048] Brain PET uses positron-emitting diode (PET) radiolabeled glucose and other human metabolites as imaging agents. It reflects metabolic changes in lesions by observing their uptake of the imaging agent, thus providing clinical biometabolic information for diseases. This offers an effective detection method for conditions such as epilepsy, brain tumors, Parkinson's disease, and Alzheimer's syndrome. Brain PET images can detect mild cognitive impairment lesions, allowing for early intervention and potentially delaying the onset of the disease, which is significant for subsequent patient rehabilitation.

[0049] In practical applications, the most effective way to control potential risk factors is to prevent dementia, cognitive decline, and MCI (mild cognitive impairment). MCI is a transitional state between normal aging and dementia. Mild cognitive impairment is characterized by a decline in memory, language, and other cognitive functions that can be detected by others, and clinical examinations will also reveal a certain degree of impairment.

[0050] In this embodiment of the invention, by providing an MCI prediction scheme based on a lightweight neural network, the model can ensure prediction accuracy while reducing the overhead of hardware computing power, thereby enabling the popularization of MCI image detection, laying the foundation for early treatment, and reducing the incidence of dementia.

[0051] For medical imaging data, informed consent-granted brain PET images of subjects can be obtained from the Alzheimer's neuroimaging dataset, which includes brain imaging data of patients diagnosed with mild cognitive impairment (MCI) and brain imaging data of healthy individuals (NC, NormalControl).

[0052] For example, brain PET images were obtained from 300 subjects with informed consent. These images included brain images of 150 patients diagnosed with mild cognitive impairment (MCI) and brain images of healthy individuals (NC). An additional 340 subjects were selected from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset as supplementary images, comprising 170 normal controls (NC) and 170 individuals with mild cognitive impairment (MCI). The dataset was then randomly divided into training, validation, and test sets in an 8:1:1 ratio.

[0053] In some embodiments, the neural network model can be a convolutional neural network model, such as a depthwise separable convolution (DSC), which consists of two parts: channel-wise convolution and point-wise convolution, used to extract image feature maps. Compared with standard convolution, it has a lower parameter count and computational cost.

[0054] In practical applications, a lightweight neural network feature fusion model, namely the lightweight model DWViT (DepthWise linear self-attention Vision Transformer), can be constructed based on sample image data and a depthwise separable convolution and linear attention mechanism. This model is the neural network model discussed in this paper. The model is then trained and tested to obtain the optimal model. In some examples, model hyperparameters can be set to train and fine-tune the DWViT model. The model with the highest accuracy on the validation set is selected, and the test set accuracy is obtained. If the test set accuracy is greater than a set threshold, this model is selected as the inference model.

[0055] For example, a network model is built using the deep learning framework PyTorch, and the model hyperparameters are set. Two NVIDIA V100 GPUs are used with a batch size of 256, epochs of 300, and the AdamW optimizer is employed. Label smoothing cross-entropy loss (smoothing = 0.1) is used, and the learning rate is adjusted from 0.0002 to 0.002 over the first 3000 iterations, then reduced to 0.0002 using periodic cosine annealing. The network model architecture parameters are set as follows. Figure 2a The details of the network model architecture are as follows: the DWViT model is trained and tuned according to the model hyperparameters set as described above. The model with the highest accuracy on the validation set is selected, and the accuracy on the test set is obtained by inputting the test set. If the accuracy on the test set is greater than the set threshold λ = 0.95, then the model is selected as the inference model.

[0056] It should be noted that the process of training, validating, and testing the neural network model using sample image data can refer to the process of predicting the target image data below, and the training, validating, and testing steps can be added on this basis. In particular, the feature extraction process can refer to the process of obtaining image depth feature vectors through multiple stages of feature extraction below, and will not be repeated here.

[0057] In some embodiments of the present invention, before training the neural network model using the sample image data, the method further includes: preprocessing the sample image data.

[0058] In some examples, data preprocessing is performed on the acquired brain PET images, including:

[0059] 1. Adjust the image data format to a format suitable for model input and denoise the images to improve image computation efficiency; 2. Perform image data augmentation through operations such as rotation, flipping, color dithering, and size scaling; 3. Process the image data so that it can be input into the model in tensor form (a tensor is a multidimensional array, which is convenient for input into the model).

[0060] Step 102: Acquire target image data and use a trained neural network model to predict the target image data to obtain the prediction result of the target image data; wherein, the trained neural network model is used to perform feature extraction on the target image data in multiple stages to obtain image depth feature vectors, perform global average pooling operation on the obtained image depth feature vectors, and obtain the predicted class probability of the target image data through a classifier after passing through a fully connected layer, and select the class with the highest predicted class probability as the prediction result of the target image data.

[0061] In some examples, the feature extraction part of the network model consists of five different feature extraction stages.

[0062] In practical applications, the target image data to be predicted is input into the optimal lightweight network inference model DWViT to obtain the prediction result. For example... Figure 2b The model extracts features from the image through five stages: Stage 1 yields a 128x128 vector; Stage 2 yields a 64x64 vector; Stage 3 yields a 32x32 vector; Stage 4 yields a 16x16 vector; and Stage 5 yields an 8x8 vector. These five stages extract the image's depth features. The feature vectors are then subjected to global average pooling, passed through a fully connected layer, and finally through a softmax layer (classifier) ​​to obtain the predicted class probabilities. The class with the highest probability is then identified as the image's class.

[0063] In some embodiments of the present invention, the plurality of stages includes a first stage, wherein the feature extraction process of the first stage includes:

[0064] The target image data is processed by ordinary convolution, and the resulting feature vector is processed by a first lightweight convolution block to obtain the feature vector output by the first stage.

[0065] The first lightweight convolutional block is processed as follows:

[0066] The input feature vector is increased in dimensionality to obtain a high-dimensional feature vector. The high-dimensional feature vector is then subjected to a depthwise convolution operation to obtain a feature map. The feature map is then subjected to an average pooling operation to obtain a feature vector with a certain number of channels. The feature vector is then passed through a fully connected layer with an activation function to obtain matrix attention vector weights. The matrix attention vector weights are then multiplied element-wise with the feature map to obtain a new feature map, which is then reduced in dimensionality. Finally, the reduced-dimensional feature vector is residually concatenated with the input feature vector to obtain the output feature vector.

[0067] In stage 1 (i.e., the first stage), it consists of a regular convolution with a kernel size of 3x3 and a lightweight convolution block Block1 (i.e., the first lightweight convolution block), wherein the structure of the lightweight convolution block Block1 is as follows: Figure 2c The feature vector is increased in dimensionality using 1x1 convolutions. The high-dimensional feature vector is then passed through a depthwise convolution to obtain a feature map R. The feature map R is first subjected to average pooling to obtain a feature vector with the number of channels. The feature vector is then passed through a fully connected layer with an activation function to obtain matrix attention vector weights. The matrix attention vector weights are then element-wise multiplied with the feature map R to obtain the feature map, which is then reduced in dimensionality using 1x1 convolutions. Finally, the dimensionality-reduced vector is residually concatenated with the input vector.

[0068] In some examples, stage 1 consists of a 3x3 regular convolution and a lightweight convolutional block BLOCK1, both with 16 channels. The regular convolution downsamples the image to extract features before inputting them into the lightweight convolutional block BLOCK1. This lightweight convolutional block BLOCK1 first passes through a 1x1 convolution to boost the 16 channels to 64 channels, and then passes through a depthwise separable convolution to obtain R. 128 ×128×64 The vector is then global pooled into R. 1×1×64 The attention weight matrix obtained through a fully connected layer and the αswish activation function is multiplied by the global pooling vector to obtain R. 128×128×64 The vector is reduced to R dimension again through 1x1 convolution. 128 ×128×16 The vector is residually concatenated with the input vector and used as the input for the next stage.

[0069] In some examples, the activation function formula in the BLOCK1 structure diagram is as follows:

[0070]

[0071] elu3=min (elu,3) (2)

[0072]

[0073] In some embodiments of the present invention, the plurality of stages includes a second stage, wherein the feature extraction process of the second stage includes:

[0074] The feature vector output from the first stage is mapped and input into the second stage, processed by the second lightweight convolutional block, and the resulting feature vector is processed by the first lightweight convolutional block to obtain the feature vector output from the second stage.

[0075] The second lightweight convolutional block is processed as follows:

[0076] The input feature map is increased in dimensionality by increasing the number of channels, and then the spatial size is reduced by depthwise convolution to obtain the feature map. The obtained feature map is then subjected to global pooling to obtain the feature vector of the number of channels. The obtained feature vector is then passed through a fully connected layer with an activation function to obtain the matrix attention vector weights. The obtained matrix attention vector weights are then multiplied element-wise with the feature map to obtain the feature map. Finally, dimensionality reduction is performed to obtain the feature vector with reduced spatial size.

[0077] In stage 2 (i.e., the second stage), it consists of a lightweight convolutional block BLOCK2 (i.e., the second lightweight convolutional block) and two lightweight convolutional blocks BLOCK1. The structure of lightweight convolutional block BLOCK2 is as follows: Figure 2dAs shown, the input feature map is first enlarged by increasing the number of channels through a 1x1 convolution, and then reduced in size by a depthwise convolution with a stride of 2, reducing it to half the size of the input space, thus obtaining the feature map. The feature map obtains a feature vector with the number of channels through global pooling, and the feature vector is then passed through a fully connected layer with an activation function to obtain matrix attention vector weights. The matrix attention vector weights are element-wise multiplied with the feature map to obtain the feature map, which is then reduced in size by a 1x1 convolution, resulting in a feature vector with a reduced spatial size. Finally, it passes through two lightweight convolutional blocks (BLOCK1) to obtain the input vector for the next stage.

[0078] In some examples, stage 2 consists of a lightweight convolutional block BLOCK2 and two lightweight convolutional blocks BLOCK1, and the output vector of stage 1 is R. 128×128×16 Using lightweight convolutional blocks like BLOCK2, Figure 2d As shown, the dimensionality is first increased from 16 to 128 using a 1x1 convolution, and then the spatial size is reduced from 128 to 64 using a depthwise separable convolution operation. The feature vector is R. °64×64×128 The attention weight vector R is obtained through global pooling and fully connected layers. 1×1×128 Weight vector and R °64×64×128 The product is used to obtain a new feature vector, which is then subjected to feature fusion and dimensionality reduction through 1x1 convolution to obtain the feature vector R. 64 ×64×24 The feature vector is then passed through two lightweight convolutional blocks (BLOCK1 blocks) to obtain the input vector R for stage 3. 64×64×24 .

[0079] In some embodiments of the present invention, the plurality of stages includes a third stage, wherein the feature extraction process of the third stage includes:

[0080] The second lightweight convolutional block is used to downsample the feature vector output by the second stage in terms of spatial size. The sampled feature vector is then input into the local-global fusion module to fuse the global and local features of the feature vector to obtain a fused feature vector. The fused feature vector is then residually connected with the feature vector output by the second stage to obtain the feature vector output by the third stage.

[0081] The local-to-global fusion module employs the following processing:

[0082] The input feature map is amplified in terms of channel number, and then the feature vector is input into the local feature extraction block and the global feature extraction block respectively to obtain the local feature vector and the global feature vector. The feature is then fused by convolution of size 1x1 to obtain the fused feature vector.

[0083] In stage 3 (i.e., the third stage), a lightweight convolutional block BLOCK2 is used to spatially downsample the feature vector output from stage 2. The sampled feature vector is then input into the Local-Global Fusion (LGFB) module, where global and local features are fused to obtain the fused feature vector of the MCI image. The LGFB architecture diagram is shown below. Figure 2e As shown, the input feature map is first augmented with 1x1 convolutions to increase the number of channels, and then the vector is input into the local feature extraction block LBLOCK and the global feature extraction block GBLOCK, respectively. The resulting local and global features are then fused using 1x1 convolutions. The fused features are then residually concatenated with the input features to obtain the output feature vector of this layer.

[0084] In some examples, the input vector for stage 3 is passed through a lightweight convolutional block BLOCK2 to obtain the spatial size R. 32×32×48 The feature vectors are used as inputs to the LGFB block, and the LGFB block is as follows: Figure 2e As shown, a 1×1 convolution will transform R 32×32×48 The input features are projected into a higher-dimensional space to obtain the feature mapping X1∈R 32×32×64 X is processed by the Global Feature Extraction Module (GBLOCK) to extract global relational features and local information from local features.

[0085] In some examples, the GBLOCK extraction module, such as Figure 2f As shown, a feature vector of size (H, W, d) is divided into N feature patches of size (h, w, d). Each (h, w, d) feature vector is then expanded to obtain N (w*h, d) vectors. These vectors are input into B ViTs with αswish linear self-attention (which reduces the inference time and memory overhead of multi-head self-attention in Transformers, making it more lightweight than ordinary ViTs) to obtain the global relationships between feature vectors. Here, B is a hyperparameter that can be set. The ViT with αswish linear self-attention outputs multiple (w*h, d) vectors, which are then stacked to restore the global relationship feature vector of size (H, W, d).

[0086] In some examples, the ViT of αswish linear self-attention is specifically expressed as follows:

[0087] x∈R N×F It is represented as a sequence of N feature vectors of dimension F. The Transformer is a function T(R) N×F →R N×F It is composed of L Transformer layers. It can be defined as...

[0088] T l(x)=f l (A l (x)+x). (4)

[0089] f l Typically implemented using a feedforward neural network, A l (x) is the self-attention mechanism function. The input sequence x∈R N×F Through three matrix vectors W Q ∈R F×D W k ∈R F×D W v ∈R F×M Project onto the corresponding Q, K, V.

[0090] Q = xW Q (5)

[0091] K = xW k (6)

[0092] Q = xW v (7)

[0093]

[0094] Equation 8 implements a specific form of self-attention called softmax attention, where the similarity score is the exponent of the dot product between query Q and key K. Given a matrix with index i returning the i-th row as a vector, we can write a generalized attention equation for any similarity function, as follows:

[0095]

[0096] when When formula (9) is non-negative, formula (9) can be defined as the attention function. Then formula (9) can be redefined as:

[0097]

[0098] Formula (10) in vector form is:

[0099]

[0100]

[0101] Substituting Equation 12 into Equation 11 yields the definition of αswish linear self-attention. This leads to the ViT block of αswish linear self-attention.

[0102] In some examples, LBLOCK blocks are like Figure 2gAs shown, the input feature vector is first subjected to a 3x3 convolution operation to obtain the feature map. Then, a 3x3 depthwise convolution is performed on each channel, and finally, a point convolution operation is performed on all channels to obtain the spatial local feature vector as the output.

[0103] In some examples, the GBLOCK extraction module first divides X1 into 256 non-overlapping parts, which are then unfolded into shapes X′∈R. 4×256×64 The input is fed into two linear Transformers with αswish linear self-attention to perform global relational modeling and obtain X. G =line_Transformer(X) ′ ),X G ∈R 4×256×64 Then X G Stacked into X local R 32×32×64 Form, X L Feature fusion is performed through a 1x1 convolutional layer to output shape X. g =R 32×32×48 Feature vectors, local feature extraction such as Figure 2g As shown, X1 is first processed through a 3x3 ordinary convolution, and then through a depthwise separable convolution to obtain the shape X. local ∈R 32×32×48 eigenvectors, X local With X g Make splicing X f ∈R 32×32×96 The feature vector X is obtained by passing it through a 1x1 convolutional layer. F As input for the next stage.

[0104] In some embodiments of the present invention, the plurality of stages includes a fourth stage, the feature extraction process of the fourth stage comprising:

[0105] The features output from the third stage are reduced in spatial dimension by a second lightweight convolutional block, and the resulting feature vector is then input into a local feature fusion block to obtain a local-global fused feature vector, which is used as the feature vector output from the fourth stage.

[0106] In stage 4 (i.e., the fourth stage), the feature maps from stage 3 are obtained. First, spatial dimensionality reduction is performed using a lightweight convolutional block (BLOCK2). Then, the features are input into the local feature fusion block (LFGB) to obtain local-global fused features, such as... Figure 2b As shown, in this stage, LFGB uses a ViT with 4 αswish linear self-attention to extract global feature relationships from the feature vector.

[0107] In some embodiments of the present invention, the plurality of stages includes a fifth stage, wherein the feature extraction process of the fifth stage includes:

[0108] The feature vector output from the fourth stage is input into the second lightweight convolutional block for spatial downsampling, and the resulting feature vector is input into the local-global fusion block to obtain the fused feature vector. The resulting fused feature vector is then passed through a convolutional layer with a convolutional size of 1x1 to obtain the image depth feature vector.

[0109] In stage 5 (i.e., the fifth stage), the feature output from stage 4 is obtained and input into the lightweight convolutional block BLOCK2 for spatial downsampling. The resulting feature vector is then input into the Local-Global Fusion Block (LGFB) to obtain fused features. This stage employs three linear self-attention (ViT) layers to extract global relationships of image features. The fused features are then processed through a 1x1 convolutional layer to obtain image depth features.

[0110] In some examples, the operations of stages 4 and 5 are similar to those of stage 3, with stage 5 outputting a feature vector X5∈R. 8 ×8×512 The final class probability P is obtained after global pooling and fully connected layers. out ∈R 1×1×2 The category with the highest probability is taken as the image prediction result.

[0111] In this embodiment of the invention, sample image data is acquired and used to train a neural network model. Target image data is then acquired, and the trained neural network model is used to predict the target image data, resulting in a prediction result. The trained neural network model performs multi-stage feature extraction on the target image data to obtain an image depth feature vector. Global average pooling is applied to the obtained image depth feature vector, and after passing through a fully connected layer, a classifier is used to obtain the predicted class probability of the target image data. The class with the highest predicted class probability is selected as the prediction result of the target image data. This optimizes the neural network model used for image analysis, reduces the hardware computing power requirements of the neural network model, and improves the accuracy of image analysis.

[0112] In some examples, embodiments of the present invention implement a lightweight and high-precision MCI prediction model. The key to ensuring the accuracy of the MCI prediction model lies in enhancing the model's feature expression, including the rich microstructure of local image regions and important regional features of the global image. The local and global features in medical images are fused to obtain a more powerful feature expression. A lightweight neural network based on depthwise separable convolution and ViT with separable attention mechanism is designed to realize MCI prediction analysis.

[0113] In some examples, embodiments of the present invention have the following advantages:

[0114] 1. Based on lightweight convolutional neural network design, two different lightweight convolutional neural network blocks are constructed. Lightweight convolutional blocks BLOCK1 and BLOCK2 have the advantages of fewer parameters, faster computation, and lower computational overhead.

[0115] 2. Design a new activation function with characteristics such as no upper bound, lower bound, smoothness, and non-monotonicity, which can enable neural network layers to have richer expressive capabilities. Based on this activation function, design a new linear attention mechanism with linear complexity, which greatly reduces the computational time and space overhead of the model.

[0116] 3. Based on a lightweight convolutional neural network and ViT with separable self-attention, a lightweight global feature extraction block GBLOCK and a lightweight feature fusion module LFGB are designed respectively. This module incorporates a linear self-attention mechanism, effectively avoiding information loss and the inherent sequential structure, providing a wider receptive field, and has the advantages of fast computation and low computational cost compared with other image Transformer models.

[0117] 4. By fusing depthwise separable convolution and lightweight ViT, it is possible to simultaneously establish local image relationships and longer-distance feature relationships, thereby improving the representation capability of image depth features.

[0118] Reference Figure 3 The diagram illustrates a flowchart of another processing method based on a neural network model according to an embodiment of the present invention, which may specifically include the following steps:

[0119] Step 301: Obtain sample image data and use the sample image data to train a neural network model.

[0120] Step 302: Preprocess the sample image data.

[0121] Step 303: Acquire target image data and use a trained neural network model to predict the target image data to obtain the prediction result of the target image data; wherein, the trained neural network model is used to perform feature extraction on the target image data in multiple stages to obtain image depth feature vectors, perform global average pooling operation on the obtained image depth feature vectors, and obtain the predicted class probability of the target image data through a classifier after passing through a fully connected layer, and select the class with the highest predicted class probability as the prediction result of the target image data.

[0122] In this embodiment of the invention, sample image data is acquired and used to train a neural network model. Target image data is then acquired, and the trained neural network model is used to predict the target image data, resulting in a prediction result. The trained neural network model performs multi-stage feature extraction on the target image data to obtain an image depth feature vector. Global average pooling is applied to the obtained image depth feature vector, and after passing through a fully connected layer, a classifier is used to obtain the predicted class probability of the target image data. The class with the highest predicted class probability is selected as the prediction result of the target image data. This optimizes the neural network model used for image analysis, reduces the hardware computing power requirements of the neural network model, and improves the accuracy of image analysis.

[0123] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0124] One embodiment of the present invention provides a processing device based on a neural network model, specifically used for:

[0125] Acquire sample image data and use the sample image data to train a neural network model;

[0126] The target image data is acquired, and a trained neural network model is used to predict the target image data to obtain the prediction result of the target image data. The trained neural network model is used to extract features from the target image data in multiple stages to obtain image depth feature vectors, perform global average pooling on the obtained image depth feature vectors, and obtain the predicted class probability of the target image data through a classifier after passing through a fully connected layer. The class with the highest predicted class probability is selected as the prediction result of the target image data.

[0127] In some embodiments of the present invention, the plurality of stages includes a first stage, wherein the feature extraction process of the first stage includes:

[0128] The target image data is processed by ordinary convolution, and the resulting feature vector is processed by a first lightweight convolution block to obtain the feature vector output by the first stage.

[0129] The first lightweight convolutional block is processed as follows:

[0130] The input feature vector is increased in dimensionality to obtain a high-dimensional feature vector. The high-dimensional feature vector is then subjected to a depthwise convolution operation to obtain a feature map. The feature map is then subjected to an average pooling operation to obtain a feature vector with a certain number of channels. The feature vector is then passed through a fully connected layer with an activation function to obtain matrix attention vector weights. The matrix attention vector weights are then multiplied element-wise with the feature map to obtain a new feature map, which is then reduced in dimensionality. Finally, the reduced-dimensional feature vector is residually concatenated with the input feature vector to obtain the output feature vector.

[0131] In some embodiments of the present invention, the plurality of stages includes a second stage, wherein the feature extraction process of the second stage includes:

[0132] The feature vector output from the first stage is mapped and input into the second stage, processed by the second lightweight convolutional block, and the resulting feature vector is processed by the first lightweight convolutional block to obtain the feature vector output from the second stage.

[0133] The second lightweight convolutional block is processed as follows:

[0134] The input feature map is increased in dimensionality by increasing the number of channels, and then the spatial size is reduced by depthwise convolution to obtain the feature map. The obtained feature map is then subjected to global pooling to obtain the feature vector of the number of channels. The obtained feature vector is then passed through a fully connected layer with an activation function to obtain the matrix attention vector weights. The obtained matrix attention vector weights are then multiplied element-wise with the feature map to obtain the feature map. Finally, dimensionality reduction is performed to obtain the feature vector with reduced spatial size.

[0135] In some embodiments of the present invention, the plurality of stages includes a third stage, wherein the feature extraction process of the third stage includes:

[0136] The second lightweight convolutional block is used to downsample the feature vector output by the second stage in terms of spatial size. The sampled feature vector is then input into the local-global fusion module to fuse the global and local features of the feature vector to obtain a fused feature vector. The fused feature vector is then residually connected with the feature vector output by the second stage to obtain the feature vector output by the third stage.

[0137] The local-to-global fusion module employs the following processing:

[0138] The input feature map is amplified in terms of channel number, and then the feature vector is input into the local feature extraction block and the global feature extraction block respectively to obtain the local feature vector and the global feature vector. The feature is then fused by convolution of size 1x1 to obtain the fused feature vector.

[0139] In some embodiments of the present invention, the plurality of stages includes a fourth stage, the feature extraction process of the fourth stage comprising:

[0140] The features output from the third stage are reduced in spatial dimension by a second lightweight convolutional block, and the resulting feature vector is then input into a local feature fusion block to obtain a local-global fused feature vector, which is used as the feature vector output from the fourth stage.

[0141] In some embodiments of the present invention, the plurality of stages includes a fifth stage, wherein the feature extraction process of the fifth stage includes:

[0142] The feature vector output from the fourth stage is input into the second lightweight convolutional block for spatial downsampling, and the resulting feature vector is input into the local-global fusion block to obtain the fused feature vector. The resulting fused feature vector is then passed through a convolutional layer with a convolutional size of 1x1 to obtain the image depth feature vector.

[0143] In some embodiments of the present invention, before training the neural network model using the sample image data, the method is further used to:

[0144] The sample image data is preprocessed.

[0145] In this embodiment of the invention, sample image data is acquired and used to train a neural network model. Target image data is then acquired, and the trained neural network model is used to predict the target image data, resulting in a prediction result. The trained neural network model performs multi-stage feature extraction on the target image data to obtain an image depth feature vector. Global average pooling is applied to the obtained image depth feature vector, and after passing through a fully connected layer, a classifier is used to obtain the predicted class probability of the target image data. The class with the highest predicted class probability is selected as the prediction result of the target image data. This optimizes the neural network model used for image analysis, reduces the hardware computing power requirements of the neural network model, and improves the accuracy of image analysis.

[0146] An embodiment of the present invention also provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the above-described processing method based on a neural network model.

[0147] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the above-described processing method based on a neural network model.

[0148] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0150] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0151] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0156] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the aforementioned element.

[0157] The above provides a detailed description of a processing method, apparatus, device, and medium based on a neural network model. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A processing method based on a neural network model, characterized in that, The method includes: Acquire sample image data and use the sample image data to train a neural network model; The target image data is acquired, and a trained neural network model is used to predict the target image data to obtain the prediction result of the target image data. The trained neural network model is used to extract features from the target image data in multiple stages to obtain an image depth feature vector, perform global average pooling on the obtained image depth feature vector, and then pass it through a classifier after passing through a fully connected layer to obtain the predicted class probability of the target image data. The class with the highest predicted class probability is selected as the prediction result of the target image data. The plurality of stages includes a first stage, the feature extraction process of which includes: The target image data is processed by ordinary convolution, and the resulting feature vector is processed by a first lightweight convolution block to obtain the feature vector output by the first stage. The first lightweight convolutional block is processed as follows: The input feature vector is increased in dimensionality to obtain a high-dimensional feature vector. The high-dimensional feature vector is then subjected to a depthwise convolution operation to obtain a feature map. The feature map is then subjected to an average pooling operation to obtain a feature vector with a certain number of channels. The feature vector is then passed through a fully connected layer with an activation function to obtain matrix attention vector weights. The matrix attention vector weights are then multiplied element-wise with the feature map to obtain a new feature map, which is then reduced in dimensionality. Finally, the reduced feature vector is residually concatenated with the input feature vector to obtain the output feature vector. The plurality of stages includes a second stage, and the feature extraction process of the second stage includes: The feature vector output from the first stage is mapped and input into the second stage, processed by the second lightweight convolutional block, and the resulting feature vector is processed by the first lightweight convolutional block to obtain the feature vector output from the second stage. The second lightweight convolutional block is processed as follows: The input feature map is increased in dimensionality by increasing the number of channels, and then the spatial size is reduced by depthwise convolution to obtain the feature map. The obtained feature map is then subjected to global pooling to obtain the feature vector of the number of channels. The obtained feature vector is then passed through a fully connected layer with an activation function to obtain the matrix attention vector weights. The obtained matrix attention vector weights are then multiplied element-wise with the feature map to obtain the feature map. Finally, dimensionality reduction is performed to obtain the feature vector with reduced spatial size. The plurality of stages includes a third stage, and the feature extraction process of the third stage includes: The second lightweight convolutional block is used to downsample the feature vector output by the second stage in terms of spatial size. The sampled feature vector is then input into the local-global fusion module to fuse the global and local features of the feature vector to obtain a fused feature vector. The fused feature vector is then residually connected with the feature vector output by the second stage to obtain the feature vector output by the third stage. The local-to-global fusion module employs the following processing: The input feature map is amplified in terms of channel number, and then the feature vector is input into the local feature extraction block and the global feature extraction block respectively to obtain the local feature vector and the global feature vector. The feature is then fused by convolution of size 1x1 to obtain the fused feature vector.

2. The method according to claim 1, characterized in that, The plurality of stages includes a fourth stage, the feature extraction process of which includes: The features output from the third stage are reduced in spatial dimension by a second lightweight convolutional block, and the resulting feature vector is then input into a local feature fusion block to obtain a local-global fused feature vector, which is used as the feature vector output from the fourth stage.

3. The method according to claim 2, characterized in that, The plurality of stages includes a fifth stage, the feature extraction process of which includes: The feature vector output from the fourth stage is input into the second lightweight convolutional block for spatial downsampling, and the resulting feature vector is input into the local-global fusion block to obtain the fused feature vector. The resulting fused feature vector is then passed through a convolutional layer with a convolutional size of 1x1 to obtain the image depth feature vector.

4. The method according to any one of claims 1 to 3, characterized in that, Before training the neural network model using the sample image data, the method further includes: The sample image data is preprocessed.

5. A processing device based on a neural network model, characterized in that, The device is used for: Acquire sample image data and use the sample image data to train a neural network model; The target image data is acquired, and a trained neural network model is used to predict the target image data to obtain the prediction result of the target image data. The trained neural network model is used to extract features from the target image data in multiple stages to obtain an image depth feature vector, perform global average pooling on the obtained image depth feature vector, and then pass it through a classifier after passing through a fully connected layer to obtain the predicted class probability of the target image data. The class with the highest predicted class probability is selected as the prediction result of the target image data. The plurality of stages includes a first stage, the feature extraction process of which includes: The target image data is processed by ordinary convolution, and the resulting feature vector is processed by a first lightweight convolution block to obtain the feature vector output by the first stage. The first lightweight convolutional block is processed as follows: The input feature vector is increased in dimensionality to obtain a high-dimensional feature vector. The high-dimensional feature vector is then subjected to a depthwise convolution operation to obtain a feature map. The feature map is then subjected to an average pooling operation to obtain a feature vector with a certain number of channels. The feature vector is then passed through a fully connected layer with an activation function to obtain matrix attention vector weights. The matrix attention vector weights are then multiplied element-wise with the feature map to obtain a new feature map, which is then reduced in dimensionality. Finally, the reduced feature vector is residually concatenated with the input feature vector to obtain the output feature vector. The plurality of stages includes a second stage, and the feature extraction process of the second stage includes: The feature vector output from the first stage is mapped and input into the second stage, processed by the second lightweight convolutional block, and the resulting feature vector is processed by the first lightweight convolutional block to obtain the feature vector output from the second stage. The second lightweight convolutional block is processed as follows: The input feature map is increased in dimensionality by increasing the number of channels, and then the spatial size is reduced by depthwise convolution to obtain the feature map. The obtained feature map is then subjected to global pooling to obtain the feature vector of the number of channels. The obtained feature vector is then passed through a fully connected layer with an activation function to obtain the matrix attention vector weights. The obtained matrix attention vector weights are then multiplied element-wise with the feature map to obtain the feature map. Finally, dimensionality reduction is performed to obtain the feature vector with reduced spatial size. The plurality of stages includes a third stage, and the feature extraction process of the third stage includes: The second lightweight convolutional block is used to downsample the feature vector output by the second stage in terms of spatial size. The sampled feature vector is then input into the local-global fusion module to fuse the global and local features of the feature vector to obtain a fused feature vector. The fused feature vector is then residually connected with the feature vector output by the second stage to obtain the feature vector output by the third stage. The local-to-global fusion module employs the following processing: The input feature map is amplified in terms of channel number, and then the feature vector is input into the local feature extraction block and the global feature extraction block respectively to obtain the local feature vector and the global feature vector. The feature is then fused by convolution of size 1x1 to obtain the fused feature vector.

6. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the processing method based on a neural network model as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the processing method based on a neural network model as described in any one of claims 1 to 4.

Images