Confocal microendoscopic image lesion diagnosis method and related equipment

Abstract

The present application discloses a confocal microendoscopic image lesion diagnosis method and related equipment, which relate to the field of image processing. The method comprises: according to a region generation module, cutting out a plurality of sub-images of different scales and sizes from different positions on an original image, wherein each of the above sub-images includes a central region of the image; inputting all sub-images into a feature extraction module to extract feature information of a plurality of sub-images; inputting all sub-image feature information into an attention generation module to obtain feature information of an attention-enhanced sub-image; and fusing the above sub-image feature information and the above attention-enhanced sub-image feature information and inputting them into a classification module to obtain an image classification result.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a confocal microendoscopic image lesion diagnosis method and related equipment. Background Art

[0002] Confocal microendoscopic images are a high-resolution, real-time imaging technology that is widely used in medical imaging diagnosis, especially in the early detection of lesion areas. In this field, the feature differences between lesion images and normal images may be very subtle, especially in the early lesion stage. This inter-class difference is difficult to be captured by traditional convolutional neural networks. Traditional methods usually take the entire image as input and may not be able to effectively focus on the local features of the lesion area, resulting in insufficient sensitivity of the model to small lesion areas. Existing network architectures lack sufficient robustness when faced with input images of different scales or different lesion types, especially when training data is limited.

[0003] Therefore, it is necessary to propose a confocal microendoscopic image lesion diagnosis method and related equipment to at least solve some of the above problems. Summary of the invention

[0004] A series of simplified concepts are introduced in the Summary of the Invention section, which will be further described in detail in the Detailed Description of the Invention section. The Summary of the Invention section of this application does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0005] In a first aspect, the present application proposes a confocal microendoscopic image lesion diagnosis method, comprising:

[0006] According to the region generation module, a plurality of sub-images of different scales and sizes are cut out from different positions on the original image, wherein each of the sub-images includes a central region of the image;

[0007] Input all sub-images into a feature extraction module to extract feature information of multiple sub-images;

[0008] Input all sub-image feature information into the attention generation module to obtain attention enhanced sub-image feature information;

[0009] The above sub-image feature information and the above attention enhanced sub-image feature information are fused and input into the classification module to obtain the image classification result.

[0010] In a feasible implementation, the training step of the above classification module includes:

[0011] Define the training environment and training parameters;

[0012] The constructed data set is input into the model to be trained based on the cross-validation method for training until the training reaches a preset number of times to complete the training of the above-mentioned classification module, wherein the above-mentioned constructed data set includes a training set, a validation set and a test set.

[0013] In a feasible implementation manner, the optimizer of the training process of the above-mentioned classification module is an adam optimizer, the momentum parameter of the above-mentioned adam optimizer is 0.9, and the weight attenuation is set to 0.005.

[0014] In a feasible implementation, the feature extraction module is an improved MobileNet_v3 network, and the improved MobileNet_v3 network is obtained by replacing DropOut of the classic MobileNet_v3 network with DropBlock.

[0015] In a feasible implementation, the attention generation module includes multiple attention extraction channels, each of which includes M energy functions, and each attention extraction channel evaluates the importance of a single neuron by calculating its energy function.

[0016] In a feasible implementation, the attention generation module divides the weights of the generated attention layers one by one based on the LSTM structure.

[0017] In a feasible implementation, the energy function is calculated based on the following formula:

[0018]

[0019] Among them, e t is the energy function of a single neuron, M is the total number of neurons on the channel, x i is the input feature value of the i-th neuron, t is the input feature value of a specific target neuron, and w t is the weight parameter, b t is the bias parameter, -1-(w t x i +b t ) and 1-(w t t+b t ) represents the deviation of the neuron output from the expected value, λ is the regularization coefficient, is the square of the weight.

[0020] In a second aspect, the present application proposes a confocal microendoscopic image lesion diagnosis device, comprising:

[0021] A region generation unit, used for cutting out a plurality of sub-images of different scales and sizes from different positions on the original image according to the region generation module, wherein each of the sub-images includes a central region of the image;

[0022] A feature extraction unit, used for inputting all sub-images into a feature extraction module to extract feature information of a plurality of sub-images;

[0023] an attention enhancement unit, used for inputting all sub-image feature information into the attention generation module to obtain attention-enhanced sub-image feature information;

[0024] The classification unit is used to fuse the above sub-image feature information and the above attention-enhanced sub-image feature information and input them into the classification module to obtain an image classification result.

[0025] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the confocal microendoscopic image lesion diagnosis method as described in any one of the first aspects above when executing the computer program stored in the memory.

[0026] In a fourth aspect, the present application further proposes a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the confocal microendoscopic image lesion diagnosis method of any one of the first aspects is implemented.

[0027] In summary, this method introduces a multi-scale cropping strategy to enhance the ability of local feature extraction. The multi-scale cropping module generates sub-images of different scales from different positions of the original image and slides and crops them with a fixed step size. At the same time, each sub-image retains the features of the central area. After the generated sub-images are adjusted to a uniform size, they are convenient for subsequent feature extraction and classification of the model. The sensitivity of the model to the local lesion area is enhanced, avoiding the problem of global feature dilution that may be caused when the entire image is input. The multi-scale cropping covers lesion areas of different sizes, improving the model's detection ability for different types of lesions. DropBlock is used instead of traditional DropOut to regularize the feature layer in the network. The generalization ability of the model under different data distributions is improved. The model's adaptability to feature diversity and uncertainty in confocal microscopy images is enhanced, especially for diagnostic scenarios with diverse lesion types and lesion degrees. A neuron importance evaluation method based on energy function is proposed. By calculating the energy value of a single neuron, the attention weight is dynamically adjusted. Compared with traditional channel or spatial attention modules, this mechanism can more flexibly enhance discriminative features such as edges and textures. More accurately capture the lesion features in confocal microscope images, especially the edge features and texture features. The sub-image features and the corresponding attention enhancement features are fused and input into the classification module. Each sub-image generates an independent classification score. Finally, the lesion diagnosis result is determined by counting all the classification results. The independent scoring mechanism of sub-images effectively reduces the missed diagnosis rate caused by the lack of attention to a single lesion area. This method achieves technological innovation in the entire process from feature extraction to classification diagnosis by combining multi-scale cropping, DropBlock regularization and a new attention generation mechanism. Compared with traditional methods, this scheme not only improves the model's lesion differentiation ability, but also enhances robustness and generalization. It is suitable for application in various types of confocal microscope image lesion diagnosis scenarios, providing a more efficient and accurate solution for medical image analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present specification. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0029] Figure 1 A schematic diagram of the process of a confocal microendoscopic image lesion diagnosis method provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of the principle of lesion diagnosis using a confocal microendoscopic image provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of an esophageal lesion provided in an embodiment of the present application;

[0032] Figure 4 A normal schematic diagram of the esophagus provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of a gastric lesion provided in an embodiment of the present application;

[0034] Figure 6 A normal schematic diagram of the stomach provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of an intestinal lesion provided in an embodiment of the present application

[0036] Figure 8 A normal schematic diagram of the intestine provided in an embodiment of the present application;

[0037] Fig. 9 A DropOut schematic diagram provided for an embodiment of the present application;

[0038] Fig.10 A schematic diagram of a DropBlock provided in an embodiment of the present application;

[0039] Fig.11 A schematic diagram for comparing recognition effects provided in an embodiment of the present application;

[0040] Fig.12 A schematic diagram of the structure of a confocal microendoscopic image lesion diagnosis device provided in an embodiment of the present application;

[0041] Fig.13 A schematic diagram of the structure of an electronic device for diagnosing lesions using a confocal microendoscopic imager is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0043] Figure 1 A schematic diagram of the process of a confocal microendoscopic image lesion diagnosis method provided in an embodiment of the present application, Figure 2 A schematic diagram of the principle of lesion diagnosis using confocal microendoscopic images provided in an embodiment of the present application, the method may specifically include:

[0044] S110, cutting out a plurality of sub-images of different scales and sizes from different positions on the original image according to the region generation module, wherein each of the sub-images includes a central region of the image;

[0045] For example, the distinguishing features of confocal normal images and lesion images are different in position and size on the image. The traditional network takes the entire image as input and only focuses on the image as a whole. The inter-class difference between confocal lesion images and normal images is sometimes very small, which is not suitable for lesion diagnosis of confocal images. Therefore, the scheme proposes a lesion diagnosis algorithm based on a multi-scale region generation module, focusing on features of different scales to improve the diagnostic performance of the model.

[0046] Multiple sub-images of different scales and sizes are cropped from different positions of the original confocal image (controlled by step size), and each sub-image includes a central region. Specifically, three sub-images of different sizes (896×896, 640×640, 384×384) can be cropped using a multi-scale strategy and slid with a specific step size (128, 192, 160). In addition, the central region is retained at each scale, and 41 sub-regions are finally generated. All sub-images are resized to a uniform size (512×512) for subsequent processing.

[0047] S120, inputting all sub-images into a feature extraction module to extract feature information of multiple sub-images;

[0048] Exemplarily, each sub-region output by the region generation module is input into the feature extraction module for feature extraction to obtain the corresponding feature layer. The feature extraction module can select the improved MobileNet_v3 network. Compared with the classic MobileNet_v3 network, the DropOut method in the network is improved and replaced by DropBlock. The ordinary DropOut only randomly blocks a part of the area, while DropBlock can block continuous area blocks. There are large differences between classes in confocal image lesion images and normal images, and the types and degrees of lesions are represented differently in the image. Using DropBlock can discard continuous areas and force the model to learn without certain features, making the model more robust to input changes, improving model robustness, and helping the model learn stronger feature representations, enhancing the performance of the model on different data, and improving model performance.

[0049] S130, inputting all sub-image feature information into an attention generation module to obtain attention enhanced sub-image feature information;

[0050] Exemplarily, the feature extraction module extracts the feature layers of each sub-region and then inputs them into the attention generation module to extract the attention layer. Compared with the traditional attention module, which can only define the importance of features by channel or spatial dimension, it limits the flexibility of attention weights that change in channels and spaces, and is not suitable for application scenarios where there are complex image representations between different categories of confocal image lesion diagnosis. Therefore, this application proposes a new attention extraction mechanism, which evaluates the importance of a single neuron by calculating its energy function. It can better focus on the distinguishing edge features and texture features between confocal image lesions, generate attention-enhanced sub-image feature information, and the attention-enhanced sub-image feature information can better distinguish different types of lesions. The lower the energy, the more important the neuron is considered to be.

[0051] S140. The sub-image feature information and the attention enhancement sub-image feature information are fused and input into a classification module to obtain an image classification result.

[0052] For example, the feature map generated by each region is fused with the corresponding attention map and then input into the classification module for classification. Each region will get a classification score, and finally all classification scores are counted, and the category with the highest score is the final classification result to determine whether there is a lesion.

[0053] In summary, this method introduces a multi-scale cropping strategy to enhance the ability of local feature extraction. The multi-scale cropping module generates sub-images of different scales from different positions of the original image and slides and crops them with a fixed step size. At the same time, each sub-image retains the features of the central area. After the generated sub-images are adjusted to a uniform size, they are convenient for subsequent feature extraction and classification of the model. The sensitivity of the model to the local lesion area is enhanced, avoiding the problem of global feature dilution that may be caused when the entire image is input. The multi-scale cropping covers lesion areas of different sizes, improving the model's detection ability for different types of lesions. DropBlock is used instead of traditional DropOut to regularize the feature layer in the network. The generalization ability of the model under different data distributions is improved. The model's adaptability to feature diversity and uncertainty in confocal microscopy images is enhanced, especially for diagnostic scenarios with diverse lesion types and lesion degrees. A neuron importance evaluation method based on energy function is proposed. By calculating the energy value of a single neuron, the attention weight is dynamically adjusted. Compared with traditional channel or spatial attention modules, this mechanism can more flexibly enhance discriminative features such as edges and textures. More accurately capture the lesion features in confocal microscope images, especially the edge features and texture features. The sub-image features and the corresponding attention enhancement features are fused and input into the classification module. Each sub-image generates an independent classification score. Finally, the lesion diagnosis result is determined by counting all the classification results. The independent scoring mechanism of sub-images effectively reduces the missed diagnosis rate caused by the lack of attention to a single lesion area. This method achieves technological innovation in the entire process from feature extraction to classification diagnosis by combining multi-scale cropping, DropBlock regularization and a new attention generation mechanism. Compared with traditional methods, this scheme not only improves the model's lesion differentiation ability, but also enhances robustness and generalization. It is suitable for application in various types of confocal microscope image lesion diagnosis scenarios, providing a more efficient and accurate solution for medical image analysis.

[0054] In some examples, the training steps of the classification module include:

[0055] Define the training environment and training parameters;

[0056] The constructed data set is input into the model to be trained based on the cross-validation method for training until the training reaches a preset number of times to complete the training of the above-mentioned classification module, wherein the above-mentioned constructed data set includes a training set, a validation set and a test set.

[0057] In some examples, the optimizer of the training process of the above classification module is an adam optimizer, the momentum parameter of the adam optimizer is 0.9, and the weight decay is set to 0.005.

[0058] For example, clinical video data of confocal examinations in hospitals in different regions at different time periods are collected, and then the videos are parsed into single-frame images. After the images with diagnostic information are screened out, professional medical personnel determine whether the screened images have lesions, obtain image labels, and finally obtain an image lesion diagnosis dataset. The dataset is mainly divided into two parts: image data and label files, such as Figures 3 to 8 shown.

[0059] The diagnostic information dataset collected a total of 61,180 clinical images from 90 patient examination data from 18 hospitals from March 20, 2022 to October 8, 2023 for annotation. The data size is shown in Table 1:

[0060] Table 1

[0061] esophagus Stomach Intestine total Lesions 10244 23528 11343 45115 normal 3998 7142 4925 16065 total 14242 30670 16268 61180

[0062] The constructed data set includes training set, validation set and test set. The training set, validation set and test set data are randomly divided with a quantity ratio of 8:1:1. Due to the imbalance of the positive and negative sample ratios, in order to ensure the balance of sample distribution, a stratified division method is adopted, that is, the lesion / normal data of each part are randomly divided according to 8:1:1. The details of the final data set division are shown in Table 2:

[0063] Table 2

[0064]

[0065] You can choose a local server for model training. The training environment can be set based on Table 3:

[0066] Table 3

[0067] Graphics RTX4090 24GB Running the system Ubuntu 18.04 frame PyTorch 1.10.0 Python Version 3.9 CUDA Version 11.1

[0068] The training parameter Epoch can be set to 150, batch_size to 16, initial learning rate to 0.001, multi-threaded data reading selection to 8, optimizer to adam, optimizer momentum parameter to 0.9, weight decay to 0.005, and weights are saved every 10 times.

[0069] The constructed data set is input into the model for training, where the training set is used to train the model, the validation set is used to verify the model performance, and the test set is used to test the model. The simple cross-validation method is selected as the training method.

[0070] First, the training samples are input into the region generation module to generate 41 sub-regions of different sizes, which are then uniformly modified to a fixed size of 512x512. Next, they are input into the feature extraction module to extract features to obtain the corresponding feature layer, which is then input into the attention extraction module to obtain the attention map corresponding to each sub-region. The LSTM method is used to evaluate the weights of different attention maps. The feature map is fused with the attention map and then input into the classification module for classification to obtain the model prediction result. The back propagation algorithm is then used to optimize the model and iterative training is performed. The model weights are saved every 10 times until 150 iterative trainings are completed.

[0071] The confusion matrix on the test set is shown in Table 4:

[0072] Table 4

[0073]

[0074] The final test accuracy is (TP+TN) / (TP+FN+FP+TN)=91.69%

[0075] The recall rate (true positive rate) recall is TP / (TP+FN) = 92.44%

[0076] Then the specificity (true negative rate) is TN / (TN+FP)=89.60%

[0077] The false positive rate is: FN / (TP+FN)=7.56%

[0078] The false negative rate is: FP / (FP+TN)=10.40%

[0079] In some examples, the feature extraction module is an improved MobileNet_v3 network, and the improved MobileNet_v3 network is obtained by replacing DropOut of the classic MobileNet_v3 network with DropBlock.

[0080] For example, MobileNet_v3 is a lightweight convolutional neural network architecture, which is mainly used for efficient image classification and feature extraction tasks on mobile devices and embedded devices. It uses the SE module (Squeeze-and-Excitation) and hard activation functions (such as Hard-Swish) to further improve performance. DropOut is a commonly used regularization method that randomly discards some neurons to reduce overfitting. DropBlock is an improved version of DropOut, which not only randomly discards a single neuron, but also discards a block of neurons adjacent to it, so that the network can better simulate the situation of local feature loss. Fig. 9The figure shows the schematic diagram of DropOut. Fig.10 This is a schematic diagram of the DropBlock principle.

[0081] The feature extraction module proposed in this method is more in line with the local characteristics of the convolutional network. It improves the robustness of the network in processing complex image features. DropBlock is introduced after each Bottleneck module or fully connected layer of MobileNet_v3. DropBlock is parameterized, such as setting the size and drop rate of the block, to ensure that the feature scale of different layers of the network is adapted. When used for real-time image processing on mobile devices or resource-constrained environments, DropBlock is used to improve the generalization ability of the model.

[0082] In some examples, the attention generation module includes multiple attention extraction channels, each of which includes M energy functions, and each attention extraction channel evaluates the importance of a single neuron by calculating its energy function.

[0083] Exemplarily, the attention generation module consists of multiple independent attention extraction channels, each channel focuses on extracting a specific feature dimension or pattern. Each channel includes M energy functions, which are used to calculate the activation energy of neurons to evaluate their importance in the input features.

[0084] The energy function can be one or more of different mechanisms, such as dot product energy, additive energy, and gating mechanisms.

[0085] Dot product energy: Calculate the dot product value of the neuron feature and the global or local weight vector.

[0086] Additive Energy: Combines the distribution of bias and activation functions to assess the importance of a neuron.

[0087] Gating mechanism: further amplify or inhibit specific neurons through the Sigmoid function.

[0088] In some examples, the attention generation module divides the weights of the generated attention layers one by one based on the LSTM structure.

[0089] Exemplarily, LSTM (Long Short-Term Memory Network) is used as the connection unit between attention layers. Each LSTM unit processes the attention weights generated by the previous layer and further adjusts the weight distribution of the current layer. After the input data passes through the initial attention generation module, a set of initial attention maps is generated. LSTM receives the attention map as input, combines its hidden state and cell state, and generates the attention map of the next layer. The output of LSTM is divided into multiple weight branches, which act on different feature channels of the network respectively.

[0090] LSTM can capture the temporal dependencies between features or the information flow between layers, generating more contextual attention weights, thus avoiding the inconsistency issues that may occur when the attention layers are optimized independently.

[0091] In some examples, the energy function is calculated based on the following formula:

[0092]

[0093] Among them, e t is the energy function of a single neuron, M is the total number of neurons on the channel, x i is the input feature value of the i-th neuron, t is the input feature value of a specific target neuron, and w t is the weight parameter, b t is the bias parameter, -1-(w t x i +b t ) and 1-(w t t+b t ) represents the deviation of the neuron output from the expected value, λ is the regularization coefficient, is the square of the weight.

[0094] Exemplarily, the feature extraction network will extract the feature layers of each sub-region and then input them into the attention generation module to extract the attention layer. Compared with the traditional attention module, which can only define the importance of features by channel or spatial dimension, it limits the flexibility of attention weights that change in channels and space, and is not suitable for application scenarios where there are complex image representations between different categories of confocal image lesion diagnosis. Therefore, the application proposes a new attention extraction mechanism, which evaluates the importance of a single neuron by calculating its energy function, so as to better focus on the distinguishing edge features and texture features between confocal image lesions, and better distinguish different types of lesions. The energy function of a single neuron is as follows:

[0095]

[0096] Among them, e t is the energy function of a single neuron, M is the total number of neurons on the channel, x i is the input feature value of the i-th neuron, t is the input feature value of a specific target neuron, and w t is the weight parameter, b t is the bias parameter, -1-(w t x i +b t ) and 1-(w t t+b t) represents the deviation of the neuron output from the expected value, λ is the regularization coefficient, is the square of the weight.

[0097] Each channel has M energy functions, respectively for w t With b t Find the partial derivative. When the first-order partial derivative is 0, it is the extreme value, as shown below:

[0098]

[0099] in, μ t With σ 2 are the mean and variance of the other neurons except the target neuron, so the minimum energy is:

[0100]

[0101] The lower the energy, the more important the neuron is.

[0102] The extraction of the attention layer can be:

[0103]

[0104] A m is the attention layer, F m is the feature layer, E is the minimum energy, ⊙ represents the matrix dot product, that is, element-by-element multiplication. In addition, in order to score each generated attention layer and prevent the generated attention maps from overlapping, the LSTM structure is selected to weight the generated attention layers one by one.

[0105] Fig.11 This is a schematic diagram of a recognition effect comparison provided in an embodiment of the present application. A comparative experiment was conducted on the constructed data set using the method adopted by this method and the traditional MobileNet_V3 network. The experimental results are as follows: Fig.11 As shown in the figure, the accuracy of the model proposed in this application on the test set is better than that of the traditional MobileNet_V3 network.

[0106] like Fig.12 As shown, the present application proposes a confocal microendoscopic image lesion diagnosis device, comprising:

[0107] A region generating unit 21, configured to cut out a plurality of sub-images of different scales and sizes from different positions on the original image according to the region generating module, wherein each of the sub-images includes a central region of the image;

[0108] A feature extraction unit 22, used for inputting all sub-images into a feature extraction module to extract feature information of a plurality of sub-images;

[0109] An attention enhancement unit 23, used for inputting all sub-image feature information into the attention generation module to obtain attention enhanced sub-image feature information;

[0110] The classification unit 24 is used to fuse the above sub-image feature information and the above attention-enhanced sub-image feature information and input them into the classification module to obtain an image classification result.

[0111] The above-mentioned confocal microendoscopic image lesion diagnosis device can also perform the following steps:

[0112] In some examples, the training steps of the classification module include:

[0113] Define the training environment and training parameters;

[0114] The constructed data set is input into the model to be trained based on the cross-validation method for training until the training reaches a preset number of times to complete the training of the above-mentioned classification module, wherein the above-mentioned constructed data set includes a training set, a validation set and a test set.

[0115] In some examples, the optimizer of the training process of the above classification module is an adam optimizer, the momentum parameter of the adam optimizer is 0.9, and the weight decay is set to 0.005.

[0116] In some examples, the feature extraction module is an improved MobileNet_v3 network, and the improved MobileNet_v3 network is obtained by replacing DropOut of the classic MobileNet_v3 network with DropBlock.

[0117] In some examples, the attention generation module includes multiple attention extraction channels, each of which includes M energy functions, and each attention extraction channel evaluates the importance of a single neuron by calculating its energy function.

[0118] In some examples, the attention generation module divides the weights of the generated attention layers one by one based on the LSTM structure.

[0119] In some examples, the energy function is calculated based on the following formula:

[0120]

[0121] Among them, e t is the energy function of a single neuron, M is the total number of neurons on the channel, x i is the input feature value of the i-th neuron, t is the input feature value of a specific target neuron, and w t is the weight parameter, bt is the bias parameter, -1-(w t x i +b t ) and 1-(w t t+b t ) represents the deviation of the neuron output from the expected value, λ is the regularization coefficient, is the square of the weight.

[0122] like Fig.13 As shown, an embodiment of the present application also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor, and when the processor 320 executes the computer program 311, the steps of any method of the above-mentioned confocal microendoscopic image lesion diagnosis are implemented.

[0123] Since the electronic device introduced in this embodiment is a device used to implement a confocal microendoscopic image lesion diagnosis device in the embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is not introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present application is within the scope of protection of this application.

[0124] During the specific implementation process, when the computer program 311 is executed by a processor, any implementation method in the embodiments corresponding to the first aspect can be implemented.

[0125] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0127] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0130] An embodiment of the present application also provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes the process of confocal microendoscopic image lesion diagnosis in the corresponding embodiment.

[0131] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0137] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A confocal microendoscopic image lesion diagnosis method, characterized in that: include: Cutting out a plurality of sub-images of different scales and sizes from different positions on the original image according to the region generation module, wherein each of the sub-images includes a central region of the image; Input all sub-images into a feature extraction module to extract feature information of multiple sub-images; Input all sub-image feature information into the attention generation module to obtain attention enhanced sub-image feature information; The sub-image feature information and the attention enhanced sub-image feature information are fused and input into a classification module to obtain an image classification result.

2. The confocal microendoscopic image lesion diagnosis method according to claim 1, characterized in that: The training steps of the classification module include: Define the training environment and training parameters; The constructed data set is input into the model to be trained based on the cross-validation method for training until the training reaches a preset number of times to complete the training of the classification module, wherein the constructed data set includes a training set, a validation set and a test set.

3. The confocal microendoscopic image lesion diagnosis method according to claim 2, characterized in that: The optimizer of the training process of the classification module is the Adam optimizer, the momentum parameter of the Adam optimizer is 0.9, and the weight decay is set to 0.

005.

4. The confocal microendoscopic image lesion diagnosis method according to claim 1, characterized in that: The feature extraction module is an improved MobileNet_v3 network, and the improved MobileNet_v3 network is obtained by replacing DropOut of the classic MobileNet_v3 network with DropBlock.

5. The confocal microendoscopic image lesion diagnosis method according to claim 1, characterized in that: The attention generation module includes multiple attention extraction channels, each of which includes M energy functions, and each attention extraction channel evaluates the importance of a single neuron by calculating its energy function.

6. According to the confocal microendoscopic image lesion diagnosis method according to claim 5, the attention generation module weights the attention layers generated one by one based on the LSTM structure.

7. The confocal microendoscopic image lesion diagnosis method according to claim 5, characterized in that: The energy function is calculated based on the following formula: Among them, e t is the energy function of a single neuron, M is the total number of neurons on the channel, x i is the input feature value of the i-th neuron, t is the input feature value of a specific target neuron, and w t is the weight parameter, b t is the bias parameter, -1-(w t x i +b t ) and 1-(w t t+b t ) represents the deviation of the neuron output from the expected value, λ is the regularization coefficient, is the square of the weight.

8. A confocal microendoscopic image lesion diagnosis device, characterized in that: include: A region generation unit, used for cutting out a plurality of sub-images of different scales and sizes from different positions on the original image according to the region generation module, wherein each of the sub-images includes a central region of the image; A feature extraction unit, used for inputting all sub-images into a feature extraction module to extract feature information of a plurality of sub-images; an attention enhancement unit, used for inputting all sub-image feature information into the attention generation module to obtain attention-enhanced sub-image feature information; A classification unit is used to fuse the sub-image feature information and the attention enhanced sub-image feature information and input them into a classification module to obtain an image classification result.

9. An electronic device, comprising: A memory and a processor, wherein the processor is used to implement the steps of the confocal microendoscopic image lesion diagnosis method as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the confocal microendoscopic image lesion diagnosis method as described in any one of claims 1 to 7 are implemented.

Images