Lung lesion recognition method and device based on multi-label classification

Abstract

The invention provides a lung lesion recognition method and device based on multi-label classification. The method comprises the steps that a to-be-detected lung medical image is acquired; respectively inputting the to-be-detected lung medical image into a first feature extraction structure and a second feature extraction structure to obtain a corresponding first feature and a corresponding second feature; performing feature extraction and fusion on the first feature and the second feature to obtain a fused feature map; and performing full connection and classification processing on the fused feature map to obtain a lesion recognition result. According to the method and the device, the multi-label classification network model is processed in two stages, so that independent training can be performed during model training, the propagation length of model training is greatly reduced, and the success probability of model training is improved.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and in particular to a method and device for identifying lung lesions based on multi-label classification. Background Art

[0002] Based on the medical images, the intelligent assessment of the human body through the model is the latest topic of AI application in medical images. However, the current intelligent assessment of lung organs is designed with large models, and the training transmission length is too long. Summary of the invention

[0003] The problem that this application solves is that the propagation length of current model training is too long.

[0004] To solve the above problems, the first aspect of the present application provides a lung lesion recognition method based on multi-label classification, comprising:

[0005] Acquire a medical image of the lung to be tested;

[0006] Inputting the lung medical image to be detected into the first feature extraction structure and the second feature extraction structure respectively to obtain corresponding first features and second features;

[0007] Extracting and fusing the first feature and the second feature to obtain a fused feature map;

[0008] The fusion feature map is fully connected and classified to obtain the lesion recognition result.

[0009] The second aspect of the present application provides a lung lesion recognition device based on multi-label classification, which includes:

[0010] An image acquisition module, which is used to acquire a lung medical image to be detected;

[0011] A feature extraction module, which is used to input the lung medical image to be detected into a first feature extraction structure and a second feature extraction structure respectively to obtain corresponding first features and second features;

[0012] A feature fusion module, which is used to extract and fuse the first feature and the second feature to obtain a fused feature map;

[0013] The lesion recognition module is used to perform full connection and classification processing on the fusion feature map to obtain the lesion recognition result.

[0014] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0015] The memory is used to store programs;

[0016] The processor, coupled to the memory, is configured to execute the program to:

[0017] Acquire a medical image of the lung to be tested;

[0018] Inputting the lung medical image to be detected into the first feature extraction structure and the second feature extraction structure respectively to obtain corresponding first features and second features;

[0019] Extracting and fusing the first feature and the second feature to obtain a fused feature map;

[0020] The fusion feature map is fully connected and classified to obtain the lesion recognition result.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned method for identifying lung lesions based on multi-label classification.

[0022] In this application, the multi-label classification network model is divided into two stages for processing, so that independent training can be performed during model training, thereby greatly reducing the propagation length of model training and increasing the probability of successful model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of a method for identifying lung lesions based on multi-label classification according to an embodiment of the present application;

[0024] Figure 2 It is an architectural diagram of a feature extraction structure of a lung lesion recognition method based on multi-label classification according to an embodiment of the present application;

[0025] Figure 3 The following is an architecture diagram of feature fusion of a lung lesion recognition method based on multi-label classification according to an embodiment of the present application;

[0026] Figure 4 It is an architecture diagram of channel attention of a lung lesion recognition method based on multi-label classification according to an embodiment of the present application;

[0027] Figure 5 is a structural block diagram of a lung lesion identification device based on multi-label classification according to an embodiment of the present application;

[0028] Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0030] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0031] In response to the above problems, the present application provides a new lung lesion identification solution based on multi-label classification, which can solve the problem of long propagation length of current model training by dividing the multi-label classification network model into two stages for processing.

[0032] The present application embodiment provides a method for identifying lung lesions based on multi-label classification. The specific scheme of the method is as follows: Figure 1-Figure 4 As shown, the method can be performed by a lung lesion recognition device based on multi-label classification, and the lung lesion recognition device based on multi-label classification can be integrated in electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 As shown, it is a flow chart of a method for identifying lung lesions based on multi-label classification according to an embodiment of the present application; wherein the method for identifying lung lesions based on multi-label classification includes:

[0033] S101, obtaining a lung medical image to be detected;

[0034] S102, inputting the to-be-detected lung medical image into a first feature extraction structure and a second feature extraction structure respectively to obtain corresponding first features and second features;

[0035] S103, extracting and fusing the first feature and the second feature to obtain a fused feature map;

[0036] S104, performing full connection and classification processing on the fused feature graph to obtain the lesion recognition result.

[0037] In this application, the processing of the multi-label classification network model is divided into two stages, the first stage is the feature extraction stage, and the second stage is the feature fusion stage to output the classification results.

[0038] Among them, in the first stage, two processing branches, a first feature extraction structure and a second feature extraction structure, are set. The input lung medical image to be detected is input into the two structures respectively, and the first feature and the second feature are output by the two structures respectively; in the second stage, the first feature and the second feature are subjected to feature fusion, full connection and classification processing, so as to obtain the lesion recognition result.

[0039] In this application, the multi-label classification network model is divided into two stages for processing, so that independent training can be performed during model training, thereby greatly reducing the propagation length of model training and increasing the probability of successful model training.

[0040] It should be noted that for deep convolutional models, if the propagation length is too long, effective information will not be transmitted during the model training process, resulting in the model being unable to iterate or converge.

[0041] The lesion identification results at least include: infectious lung disease, obstructive lung disease, tumor lung disease and restrictive lung disease.

[0042] Infectious lung diseases:

[0043] Pneumonia: caused by bacteria, viruses, fungi, etc., such as bacterial pneumonia, viral pneumonia, and pneumonia caused by new coronavirus infection.

[0044] Tuberculosis: caused by infection with Mycobacterium tuberculosis.

[0045] Lung abscess: A pus-filled cavity forms in the lung tissue after infection.

[0046] Obstructive lung disease:

[0047] Chronic obstructive pulmonary disease (COPD): includes chronic bronchitis and emphysema.

[0048] Asthma: Reversible airflow limitation caused by inflammation of the airways.

[0049] Bronchiectasis: Abnormal dilation of the bronchi with recurrent infections.

[0050] Tumor lung disease:

[0051] Lung cancer: includes small cell lung cancer (SCLC) and non-small cell lung cancer (NSCLC).

[0052] Benign tumors: such as pulmonary hamartoma.

[0053] Restrictive lung disease:

[0054] Interstitial lung disease (ILD): such as idiopathic pulmonary fibrosis (IPF) and hypersensitivity pneumonitis.

[0055] Pneumoconiosis: such as silicosis and coal workers' pneumoconiosis, caused by long-term inhalation of dust.

[0056] Pleural diseases: such as pleural thickening and pleurisy.

[0057] In one embodiment, in combination Figure 3 As shown, the step S103, extracting and fusing the first feature and the second feature to obtain a fused feature map, includes:

[0058] Perform convolution processing on feature one to obtain a convolution feature map;

[0059] The convolutional feature 2 is subjected to four-branch convolution processing to obtain the first convolution map, the second convolution map, the third convolution map and the fourth convolution map;

[0060] Multiply the convolution feature map with the first convolution map, the second convolution map, and the third convolution map respectively, and add the multiplication result to the fourth convolution map to obtain a feature extraction map;

[0061] Process the splicing coefficients of feature one and feature two to obtain the splicing coefficients;

[0062] Multiply the concatenation coefficient with the feature extraction map to obtain the fused feature map.

[0063] In this application, the convolved feature 2 is input into four parallel dilated convolutions with dilation rates of 1, 2, 3, and 1 to extract multi-scale features in feature 2.

[0064] In this application, feature one is processed using 3×3 convolution to generate a corresponding guide feature (convolution feature map); and feature two is processed using 1×1 convolution to perform four branch processing on the generated corresponding guide feature.

[0065] In the present application, the convolution feature 2 is subjected to four-branch convolution processing to obtain a first convolution map, a second convolution map, a third convolution map, and a fourth convolution map; the convolution feature map is multiplied by the first convolution map, the second convolution map, and the third convolution map, respectively, and the multiplication result is added to the fourth convolution map to obtain a feature extraction map; specifically:

[0066] Perform convolution processing on feature 2 to obtain feature 1; perform convolution processing on feature 1 with four branches to obtain corresponding features 2 / 3 / 4 / 5 respectively; perform convolution processing on feature 1 to obtain feature 6; multiply feature 2 and feature 6 to obtain feature 7, multiply feature 3 and feature 6 to obtain feature 8, multiply feature 4 and feature 6 to obtain feature 9, multiply feature 5 and feature 6 to obtain feature 10; add features 7, 8, 9, and 10 to obtain the feature extraction map.

[0067] In one embodiment, in combination Figure 3As shown, the splicing coefficient processing of feature one and feature two to obtain the splicing coefficient includes:

[0068] Perform maximum pooling and convolution processing on feature 2 to obtain the second pooling map;

[0069] Perform average pooling and convolution processing on feature one to obtain the first pooling map;

[0070] After splicing the first pooling map and the second pooling map, the convolution module is input to obtain a convolution splicing map;

[0071] The convolution splicing graph is subjected to Sigmoid activation processing to obtain the splicing coefficient.

[0072] In this application, max pooling is a downsampling operation used to reduce the size of feature maps while retaining the most important information.

[0073] In this application, the pooled image is processed by a convolutional layer, which can extract more spatial features.

[0074] In this application, average pooling performs downsampling by calculating the average value of the pooling area.

[0075] In this application, the first pooling map and the second pooling map are spliced ​​in the channel dimension, and the splicing operation can fuse features from different pooling methods; the spliced ​​feature map will be input into the subsequent convolution module, and the convolution module will further extract the high-order features of these spliced ​​features.

[0076] In this application, average pooling and maximum pooling are used through different branches to retain more features of feature one and feature two.

[0077] Preferably, after average pooling / maximum pooling, attention vector extraction is also performed to achieve maximum retention of feature one and feature two.

[0078] In this application, the convolution module includes convolution, batch normalization and ReLU activation function processing.

[0079] In this application, after splicing and convolution of feature 1 and feature 2, the corresponding splicing coefficient is obtained. Then, through the splicing coefficient, the retained features of feature 1 and feature 2 are integrated and multiplied and fused with the feature extraction map to achieve the response between the features.

[0080] In one embodiment, in combination Figure 2As shown, the first feature extraction structure and the second feature extraction structure are UNet models; when the first feature extraction structure is downsampled, the feature map of the corresponding level obtained is processed with the feature map of the corresponding level of the second feature extraction structure through a fusion module, and then downsampled to obtain the feature map of the next level of the first feature extraction structure.

[0081] Among them, feature one is obtained through the first feature extraction structure, and feature two is obtained through the second feature extraction structure; wherein the second feature extraction structure first performs downsampling to obtain multiple levels of feature maps of downsampling in sequence, and then performs upsampling based on long connections to obtain multiple levels of feature maps of upsampling in sequence, and obtains feature two through convolution.

[0082] Among them, the first feature extraction structure is similar to the second feature extraction structure, and downsampling and upsampling are performed in sequence. The difference is that when the first feature extraction structure is downsampling, the feature map of the corresponding level obtained will be processed by the fusion module with the feature map of the corresponding level of the second feature extraction structure, and then downsampled to obtain the feature map of the next level.

[0083] Specifically: the first feature extraction structure downsamples the input lung medical image to be detected once to obtain a feature map of the first level (called a first-level feature map); at the same time, the first-level feature map of the downsampled second feature extraction structure is input into the fusion module for processing to obtain the second-level feature map of the first feature extraction structure; sequentially, the second (third) level feature map of the downsampled second feature extraction structure is input into the fusion module for processing to obtain the third (fourth) level feature map of the first feature extraction structure; finally, the fourth level feature map of the downsampled second feature extraction structure is convoluted and then upsampled to obtain the upsampled feature map of the first feature extraction structure, and after subsequent multiple upsampling and convolution, feature one is obtained.

[0084] It should be noted that both inputs of the fusion module are down-sampled inputs, that is, down-sampling is performed first and then input into the fusion module.

[0085] In this application, the feature Figure 2 Extract information from the process to assist features Figure 1 The generation of features can accelerate the feature extraction process of the network.

[0086] In one embodiment, the second feature extraction structure independently performs model training, and the loss function during the training process is:

[0087]

[0088] Where H and W represent the height and width of the image respectively, X represents the input image, O represents the output image, i represents the row in the image, and j represents the column in the image.

[0089] In one embodiment, during the training process, the first feature extraction structure requires the cooperation of each downsampled feature map in the second feature extraction structure; therefore, after the training of the second feature extraction structure is completed, the first feature extraction structure and the second feature extraction structure are subjected to model training as a whole. During the training process, the parameters of the second feature extraction structure remain unchanged, and only the parameters of the first feature extraction structure and the fusion module are changed.

[0090] It should be noted that the training process and sample acquisition of the first feature extraction structure and the second feature extraction structure will not be described in detail in this application. Among them, the annotations corresponding to feature one and feature two can be intermediate feature maps obtained by experimental methods (specifically, a few can be generated by experimental methods, and then multiple sets of training data can be obtained through data enhancement).

[0091] Preferably, during the overall training process of the first feature extraction structure and the second feature extraction structure, the loss function of the second feature extraction structure for independent model training is still used, thereby increasing the homology between the first feature extraction structure and the second feature extraction structure.

[0092] In one embodiment, in combination Figure 4 As shown, the processing process of the fusion module is:

[0093] The first feature map of the input is processed by average pooling, full connection, Relu activation, full connection, and Softmax function to generate complementary first weights and second weights;

[0094] Performing continuous convolution processing on the input first feature map and multiplying it with the first weight to obtain a first weight map;

[0095] Performing continuous convolution processing on the input second feature map and multiplying it with the second weight to obtain a second weight map;

[0096] After adding the first weight map and the second weight map, an output feature map is obtained.

[0097] In the present application, the first feature map is subjected to continuous convolution processing, aiming to allow the network to focus on different features of the first feature map, thereby enhancing the weights of important features and weakening unimportant features.

[0098] In the present application, the second feature map is subjected to continuous convolution processing, aiming to allow the network to focus on different features of the second feature map, thereby enhancing the weights of important features and weakening unimportant features.

[0099] In this application, the normalized attention weights of the two branches are generated through average pooling, full connection, Relu activation, full connection, and Softmax function processing, and the global average pool is used to increase the receptive field, which allows the above processing to capture features from the entire image.

[0100] In this application, if the input features change, the weights of the two branches will also change; by limiting the weights, the learning of the entire fusion module is promoted, and applicable and adjustable attention adjustment is achieved.

[0101] In the present application, some unimportant features of interest can be eliminated through the fusion module to achieve a balance between the first feature map branch and the second feature map branch.

[0102] In the present application, the fusion feature graph is fully connected and classified, and the classification result (lesion recognition result) is output. The classification result is output through the fully connected layer and the Softmax layer.

[0103] Preferably, the classification result is output through a fully connected layer and a Softmax-like operation.

[0104] In one implementation, the activation function of the SoftMax-like operation is:

[0105]

[0106] Among them, SoftMax1(x) is the activation function, x i 、x j is the element in the input vector, i and j are the element numbers.

[0107] In this application, SoftMax is a mathematical function that is usually used to convert a set of arbitrary real numbers into real numbers representing probability distributions. It is essentially a normalization function that can convert a set of arbitrary real values ​​into probability values ​​between [0,1], because SoftMax converts them to values ​​between 0 and 1, so they can be interpreted as probabilities. If one of the inputs is small or negative, SoftMax turns it into a small probability, and if the input is large, it turns it into a large probability, but it will always remain between 0 and 1.

[0108] However, for the standard SoftMax function, since the input is mapped between 0 and 1, and the sum of all output values ​​is 1, this means that even if some input values ​​are very small, they will have a non-zero output value after being processed by the SoftMax function. This will also cause the noise to be amplified, resulting in the final output result being affected by greater noise.

[0109] In this application, a 1 is added to the denominator of the SoftMax-like function; this change means that when the input value is very small, their output value can be closer to zero. This allows the corresponding output to tend to zero when there is no valuable information to add, thus greatly reducing unnecessary noise.

[0110] An embodiment of the present application provides a lung lesion recognition device based on multi-label classification, which is used to execute the lung lesion recognition method based on multi-label classification described in the above content of the present application. The lung lesion recognition device based on multi-label classification is described in detail below.

[0111] like Figure 5 As shown, the lung lesion recognition device based on multi-label classification includes:

[0112] An image acquisition module 101 is used to acquire a lung medical image to be detected;

[0113] A feature extraction module 102, which is used to input the lung medical image to be detected into a first feature extraction structure and a second feature extraction structure respectively to obtain corresponding first features and second features;

[0114] A feature fusion module 103, which is used to extract and fuse the first feature and the second feature to obtain a fused feature map;

[0115] The lesion identification module 104 is used to perform full connection and classification processing on the fused feature map to obtain the lesion identification result.

[0116] In one embodiment, the lesion identification results include at least infectious lung disease, obstructive lung disease, neoplastic lung disease and restrictive lung disease.

[0117] In one implementation, the feature fusion module 103 is further used to:

[0118] Perform convolution processing on feature one to obtain a convolution feature map; perform convolution processing on feature two after convolution with four branches to obtain the first convolution map, the second convolution map, the third convolution map and the fourth convolution map; multiply the convolution feature map with the first convolution map, the second convolution map and the third convolution map respectively, and add the multiplication result to the fourth convolution map to obtain a feature extraction map; perform splicing coefficient processing on feature one and feature two to obtain a splicing coefficient; multiply the splicing coefficient with the feature extraction map to obtain a fused feature map.

[0119] In one implementation, the feature fusion module 103 is further used to:

[0120] Performing maximum pooling and convolution processing on feature two to obtain a second pooling map; performing average pooling and convolution processing on feature one to obtain a first pooling map; splicing the first pooling map and the second pooling map, inputting the convolution module to obtain a convolution splicing map; performing Sigmoid activation processing on the convolution splicing map to obtain the splicing coefficient.

[0121] In one embodiment, the first feature extraction structure and the second feature extraction structure are UNet models; when the first feature extraction structure is downsampled, the feature map of the corresponding level obtained is processed with the feature map of the corresponding level of the second feature extraction structure through a fusion module, and then downsampled to obtain the feature map of the next level of the first feature extraction structure.

[0122] In one embodiment, the feature extraction module 102 is further used to:

[0123] The first input feature map is processed by average pooling, full connection, Relu activation, full connection, and Softmax function to generate complementary first weights and second weights; the first input feature map is continuously convolved and multiplied with the first weight to obtain a first weight map; the second input feature map is continuously convolved and multiplied with the second weight to obtain a second weight map; the first weight map and the second weight map are added to obtain the output feature map.

[0124] In one implementation, the second feature extraction structure independently performs model training, and the loss function during the training process is:

[0125]

[0126] Where H and W represent the height and width of the image respectively, X represents the input image, O represents the output image, i represents the row in the image, and j represents the column in the image.

[0127] The lung lesion identification device based on multi-label classification provided in the above-mentioned embodiment of the present application has a corresponding relationship with the lung lesion identification method based on multi-label classification provided in the embodiment of the present application. Therefore, the specific content in the device has a corresponding relationship with the modal fusion method. The specific content can refer to the records in the modal fusion method, and will not be repeated in this application.

[0128] The lung lesion identification device based on multi-label classification provided in the above-mentioned embodiments of the present application and the lung lesion identification method based on multi-label classification provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0129] The above describes the internal functions and structure of the lung lesion recognition device based on multi-label classification, such as Figure 6As shown, in practice, the lung lesion identification device based on multi-label classification can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0130] The memory 301 may be configured to store programs.

[0131] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0132] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0133] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:

[0134] Acquire a medical image of the lung to be tested;

[0135] Inputting the lung medical image to be detected into the first feature extraction structure and the second feature extraction structure respectively to obtain corresponding first features and second features;

[0136] Extracting and fusing the first feature and the second feature to obtain a fused feature map;

[0137] The fusion feature map is fully connected and classified to obtain the lesion recognition result.

[0138] In one embodiment, the lesion identification results include at least infectious lung disease, obstructive lung disease, neoplastic lung disease and restrictive lung disease.

[0139] In one implementation, the processor 303 is further configured to:

[0140] Perform convolution processing on feature one to obtain a convolution feature map; perform convolution processing on feature two after convolution with four branches to obtain the first convolution map, the second convolution map, the third convolution map and the fourth convolution map; multiply the convolution feature map with the first convolution map, the second convolution map and the third convolution map respectively, and add the multiplication result to the fourth convolution map to obtain a feature extraction map; perform splicing coefficient processing on feature one and feature two to obtain a splicing coefficient; multiply the splicing coefficient with the feature extraction map to obtain a fused feature map.

[0141] In one implementation, the processor 303 is further configured to:

[0142] Performing maximum pooling and convolution processing on feature two to obtain a second pooling map; performing average pooling and convolution processing on feature one to obtain a first pooling map; splicing the first pooling map and the second pooling map, inputting the convolution module to obtain a convolution splicing map; performing Sigmoid activation processing on the convolution splicing map to obtain the splicing coefficient.

[0143] In one embodiment, the first feature extraction structure and the second feature extraction structure are UNet models; when the first feature extraction structure is downsampled, the feature map of the corresponding level obtained is processed with the feature map of the corresponding level of the second feature extraction structure through a fusion module, and then downsampled to obtain the feature map of the next level of the first feature extraction structure.

[0144] In one implementation, the processor 303 is further configured to:

[0145] The first input feature map is processed by average pooling, full connection, Relu activation, full connection, and Softmax function to generate complementary first weights and second weights; the first input feature map is continuously convolved and multiplied with the first weight to obtain a first weight map; the second input feature map is continuously convolved and multiplied with the second weight to obtain a second weight map; the first weight map and the second weight map are added to obtain the output feature map.

[0146] In one implementation, the second feature extraction structure independently performs model training, and the loss function during the training process is:

[0147]

[0148] Where H and W represent the height and width of the image respectively, X represents the input image, O represents the output image, i represents the row in the image, and j represents the column in the image.

[0149] In the present application, the processor is also specifically used to execute all the processes and steps of the above-mentioned lung lesion identification method based on multi-label classification. The specific content can be referred to the record in the modal fusion method, and will not be repeated in this application.

[0150] In this application, Figure 6 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 6 Components shown.

[0151] The electronic device provided in this embodiment is based on the same inventive concept as the lung lesion identification method based on multi-label classification provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored in it.

[0152] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0153] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0156] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0157] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of a computer-readable medium.

[0158] The present application also provides a computer-readable storage medium corresponding to the lung lesion identification method based on multi-label classification provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the lung lesion identification method based on multi-label classification provided in any of the aforementioned embodiments.

[0159] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0160] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the lung lesion identification method based on multi-label classification provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0161] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0162] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0163] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for identifying lung lesions based on multi-label classification, characterized in that: include: Acquire a medical image of the lung to be detected; Inputting the lung medical image to be detected into the first feature extraction structure and the second feature extraction structure respectively to obtain corresponding first features and second features; Extracting and fusing the first feature and the second feature to obtain a fused feature map; The fusion feature map is fully connected and classified to obtain the lesion recognition result.

2. The method for identifying lung lesions based on multi-label classification according to claim 1, characterized in that: The lesion identification results at least include: infectious lung disease, obstructive lung disease, tumor lung disease and restrictive lung disease.

3. The method for identifying lung lesions based on multi-label classification according to claim 1, characterized in that: The extracting and fusing the first feature and the second feature to obtain a fused feature map includes: Perform convolution processing on feature one to obtain a convolution feature map; The convolutional feature 2 is subjected to four-branch convolution processing to obtain the first convolution map, the second convolution map, the third convolution map and the fourth convolution map; Multiply the convolution feature map with the first convolution map, the second convolution map, and the third convolution map respectively, and add the multiplication result to the fourth convolution map to obtain a feature extraction map; Process the splicing coefficients of feature one and feature two to obtain the splicing coefficients; Multiply the concatenation coefficient with the feature extraction map to obtain the fused feature map.

4. The method for identifying lung lesions based on multi-label classification according to claim 3, characterized in that: The processing of the splicing coefficients of the feature one and the feature two to obtain the splicing coefficients includes: Perform maximum pooling and convolution processing on feature 2 to obtain the second pooling map; Perform average pooling and convolution processing on feature one to obtain the first pooling map; After splicing the first pooling map and the second pooling map, the convolution module is input to obtain a convolution splicing map; The convolution splicing graph is subjected to Sigmoid activation processing to obtain the splicing coefficient.

5. The method for identifying lung lesions based on multi-label classification according to any one of claims 1 to 4, characterized in that: The first feature extraction structure and the second feature extraction structure are UNet models; when the first feature extraction structure is downsampled, the feature map of the corresponding level obtained is processed with the feature map of the corresponding level of the second feature extraction structure through a fusion module, and then downsampled to obtain the feature map of the next level of the first feature extraction structure.

6. The method for identifying lung lesions based on multi-label classification according to claim 5, characterized in that: The processing process of the fusion module is as follows: The first feature map of the input is processed by average pooling, full connection, Relu activation, full connection, and Softmax function to generate complementary first weights and second weights; Performing continuous convolution processing on the input first feature map and multiplying it with the first weight to obtain a first weight map; Performing continuous convolution processing on the input second feature map and multiplying it with the second weight to obtain a second weight map; After adding the first weight map and the second weight map, an output feature map is obtained.

7. The method for identifying lung lesions based on multi-label classification according to claim 5, characterized in that: The second feature extraction structure performs model training independently, and the loss function during the training process is: Where H and W represent the height and width of the image respectively, X represents the input image, O represents the output image, i represents the row in the image, and j represents the column in the image.

8. A lung lesion recognition device based on multi-label classification, characterized in that: include: An image acquisition module, which is used to acquire a lung medical image to be detected; A feature extraction module, which is used to input the lung medical image to be detected into a first feature extraction structure and a second feature extraction structure respectively to obtain corresponding first features and second features; A feature fusion module, which is used to extract and fuse the first feature and the second feature to obtain a fused feature map; The lesion recognition module is used to perform full connection and classification processing on the fusion feature map to obtain the lesion recognition result.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Acquire a medical image of the lung to be detected; Inputting the lung medical image to be detected into the first feature extraction structure and the second feature extraction structure respectively to obtain corresponding first features and second features; Extracting and fusing the first feature and the second feature to obtain a fused feature map; The fusion feature map is fully connected and classified to obtain the lesion recognition result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the lung lesion recognition method based on multi-label classification as described in any one of claims 1 to 7.

Images