Child pneumonia classification system and method based on hierarchical multi-scale feature fusion

Abstract

The application discloses a kind of based on layered multi-scale feature fusion network's children pneumonia classification system and method, set up multiple levels to extract different scale feature map, on global branch, extract the global feature map of children chest radiograph image based on window multi-head self-attention mechanism, on local branch, extract the local feature map of children chest radiograph image using convolution operation, in fusion branch, the global feature map, local feature map extracted by each level and the fused feature map of previous level are fused, then classification is carried out again, and the classification result of children pneumonia is obtained.Extract the global feature map, local feature map of different scale, and fuse the global feature map, local feature map of current level with the fused feature map of last level, so that the feature information extracted by each level can interact, and through the feature extraction and interaction of multiple levels, the features of children chest radiograph image can be fully extracted, and the accuracy of classification result is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical artificial intelligence, in particular to a child pneumonia classification system and method based on a hierarchical multi-scale feature fusion network. BACKGROUND

[0002] Child pneumonia is divided into viral pneumonia, bacterial pneumonia, mycoplasma pneumonia and other types according to the pathogenic cause. Chest X-ray examination has the advantages of painless, non-invasive and lower radiation dose, so the X-ray chest film of children (referred to as: child chest film) is regarded as an important tool for pediatric doctors to diagnose pneumonia. However, compared with adult pneumonia diagnosis, artificial diagnosis of child pneumonia is challenged by many factors. First, children under the age of 5 have poor compliance and communication skills, and it is difficult for them to perform correct chest film examination according to the doctor's instructions; second, the bones and tissues of children have not fully developed, and there are differences in certain parts compared with adult X-ray chest film; third, the imaging performance of children of different ages on the X-ray chest film is diversified. These factors can lead to poor imaging quality of the child chest film, thereby greatly challenging the reliability and stability of the doctor's diagnosis. In addition, viewing and diagnosing X-ray chest films requires professional radiologists to complete, and manual film reading takes a long time. Therefore, computer-aided detection is needed to analyze the child chest film to assist the doctor in making a quick decision.

[0003] The existing pneumonia classification model fails to fully extract the features of the child chest film when extracting the features of the child chest film, resulting in an inaccurate child pneumonia classification result.

[0004] Therefore, the prior art needs to be improved and improved. SUMMARY

[0005] The main purpose of the present application is to provide a child pneumonia classification system based on hierarchical multi-scale feature fusion and a child chest film image classification method based on hierarchical multi-scale feature fusion, which aims to solve the problem that the features of the child chest film are not fully extracted, resulting in an inaccurate child pneumonia classification result.

[0006] In order to achieve the above purpose, the present application provides a child pneumonia classification system based on hierarchical multi-scale feature fusion, which comprises:

[0007] a chest film feature extraction module and a classification module;

[0008] The chest film feature extraction module is used to extract the features of the child chest film image, and is provided with a global branch, a local branch and a fusion branch with the same number of levels;

[0009] The global branch is configured to extract global features of the child chest image at different scales, each level of the global branch is provided with a global feature block, the global feature block is configured to extract a global feature map of the child chest image based on a window multi-head self-attention mechanism, and the scales of the global feature maps output by all global feature blocks are different from each other;

[0010] The local branch is configured to extract local features of the child chest image at different scales, each level of the local branch is provided with a local feature block, the local feature block is configured to extract a local feature map of the child chest image by using a convolution operation, and the scales of the local feature maps and the global feature maps at the same level are the same;

[0011] Each level of the fusion branch is provided with a multi-scale feature fusion block, the multi-scale feature fusion block is configured to fuse the global feature map and the local feature map at the level where the multi-scale feature fusion block is located, and the fused feature map output by the multi-scale feature fusion block at the previous level.

[0012] The classification module is configured to obtain a child pneumonia classification result according to the features output by the chest feature extraction module.

[0013] Optionally, the global feature block is provided with a first attention module using a window multi-head self-attention mechanism and a second attention module using a shift window multi-head self-attention mechanism.

[0014] Optionally, the local feature block is provided with a first convolution module and a second convolution module, the first convolution module uses a depth convolution with a size of 3*3, and the second convolution module uses a group convolution with a size of 1*1.

[0015] Optionally, the classification module is a multi-modal fusion module, the multi-modal fusion module is configured to multi-modal fuse the features of the child chest image and the features of the blood protein information output by the chest feature extraction module, and obtain a child pneumonia classification result.

[0016] Optionally, the multi-scale feature fusion block includes a fusion module and a residual inverted multi-layer perception module, the fusion module is configured to fuse the global feature map and the local feature map respectively output by the global feature block and the local feature block at the level where the multi-scale feature fusion block is located, and the fused feature map output by the multi-scale feature fusion block at the previous level, and input the obtained fused feature map into the multi-layer perception module.

[0017] Optionally, the fusion module is provided with a channel attention branch and a spatial attention branch, the channel attention branch is configured to extract features of the global feature map output by the global feature block by using a channel attention mechanism, and the spatial attention branch is configured to extract features of the local feature map output by the local feature block by using a spatial attention mechanism.

[0018] Optionally, the fusion module includes a first fusion unit and a second fusion unit. The first fusion unit is used to connect the global feature map output by the global feature block, the local feature map output by the local feature block, and the fused feature map output by the previous level multi-scale feature fusion block. The second fusion unit is used to connect the features output by the channel attention branch, the features output by the spatial attention branch, and the features output by the first fusion unit.

[0019] Optionally, the residual-inverted multilayer perceptron module includes a first convolutional module, a second convolutional module, and a third convolutional module connected in sequence. The first convolutional module uses a 3*3 depthwise convolution, the second convolutional module uses four 1*1 depthwise convolutions, and the third convolutional module uses a 1*1 depthwise convolution.

[0020] Optionally, a data preprocessing module is also provided for image preprocessing of children's chest X-ray images and data cleaning of blood protein information.

[0021] As described above, this invention establishes multiple levels to extract feature maps at different scales. In the global branch, a window-based multi-head self-attention mechanism is used to extract global feature maps from pediatric chest X-ray images. In the local branch, convolution operations are used to extract local feature maps from pediatric chest X-ray images. In the fusion branch, the global and local feature maps extracted from each level, along with the fused feature map from the previous level, are fused together before classification to obtain the classification result for pediatric pneumonia. By extracting global and local feature maps at different scales and fusing the current level's global and local feature maps with the fused feature map from the previous level, the feature information extracted at each level can interact. Furthermore, through feature extraction and interaction across multiple levels, the features of pediatric chest X-ray images can be fully extracted, improving the accuracy of the classification results.

[0022] To achieve the above objectives, the present invention also provides a classification method for pediatric chest X-ray images based on hierarchical multi-scale feature fusion, the classification method comprising:

[0023] Global and local feature maps of children's chest X-ray images are extracted at multiple levels, and the scales of the feature maps extracted at all levels are different. When extracting the global feature map, a window-based multi-head self-attention mechanism is used, and when extracting the local feature map, a convolution operation is used.

[0024] The global feature map, the local feature map, and the fused feature map of the previous level are fused together to obtain the fused feature map of each level.

[0025] Linear classification is performed on the fused feature maps of the final layer to obtain the classification results of the children's chest X-ray images.

[0026] From the above, the global feature map and the local feature map of the child chest image are extracted at multiple levels and multiple scales, and the global feature map and the local feature map of the current level are fused with the fused feature map of the last level. The feature information extracted at each level can be interacted, and through the feature extraction and interaction of multiple levels, the features of the child chest image can be fully extracted, and the accuracy of the classification result is improved. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0028] Figure 1 is the architecture schematic diagram of the child pneumonia classification system based on hierarchical multi-scale feature fusion provided by the embodiments of the present application;

[0029] Figure 2 is Figure 1 is the architecture schematic diagram of the multi-scale feature fusion block in the embodiments;

[0030] Figure 3 is Figure 1 is the comparison schematic diagram of the multi-scale feature fusion block and the ResNet, Swin Transformer and ConvNeXt network architecture in the embodiments;

[0031] Figure 4 is the flow schematic diagram of the child chest image classification method based on hierarchical multi-scale feature fusion network provided by the embodiments of the present application. DETAILED DESCRIPTION

[0032] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application, but these specific details do not limit the present application. Those skilled in the art should understand that in other embodiments, the present application can also be implemented without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted so as not to obscure the description of the present application with unnecessary details.

[0033] It should be understood that when used in the specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or sets thereof.

[0034] It should also be understood that the terms used in the specification of the present application are for the purpose of describing particular embodiments only and do not intend to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0035] It should be further understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0036] As used in the specification and the appended claims of the present application, the term "if" can be interpreted as "when" or "upon" or "in response to a determination" or "in response to detecting" depending on the context. Similarly, the phrases "if determined" or "if detected [the described condition or event]" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0037] The technical solutions in the embodiments of the present application are described clearly and completely in the following description of the embodiments of the present application in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0038] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can be practiced in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the scope of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0039] Due to the difference between the X-ray chest film of children and the X-ray chest film of adults, the existing pneumonia classification model does not consider the diversity of the feature information of the child chest film when extracting the features of the child chest film, and does not extract the features of the child chest film from multiple levels and multiple scales, that is, the features of the child chest film are not fully and accurately extracted, resulting in an inaccurate classification result of child pneumonia.

[0040] To solve the above problems, the application provides a child pneumonia classification system based on hierarchical multi-scale feature fusion, which extracts global features and local features at different scales by setting multiple levels, and fuses the global features, local features and features after fusion of the previous level at each level, so that the feature information extracted at each level can interact, and through the feature extraction and interaction of multiple levels, the features of the child chest image can be fully extracted, and the accuracy of the classification result is improved.

[0041] Exemplary system

[0042] The embodiment of the application provides a child pneumonia classification system based on hierarchical multi-scale feature fusion, which can be deployed on electronic devices such as mobile terminals, computers, notebooks and servers, and is used for analyzing child chest images and obtaining the classification result of child pneumonia, and can also be used for classifying adult chest images.

[0043] The classification system mainly comprises a chest feature extraction module and a classification module. The chest feature extraction module and the classification module are trained in advance, then the child chest image is input into the chest feature extraction module to extract the feature information of the child chest image, the feature information of the child chest image is input into the classification module for classification, and the classification result of child pneumonia can be obtained.

[0044] The chest feature extraction module comprises global branches, local branches and fusion branches with the same number of levels. Figure 1 The upper part is the global branch, the lower part is the local branch, and the middle part is the fusion branch. The levels of the global branch, the local branch and the fusion branch are all four (Stage 1-4), and the scales of each level are different, for example, the scale of the first level is 56*56*96, the scale of the second level is 28*28*192, etc. It should be noted that the number of levels is not limited to four, and can be changed accordingly as needed.

[0045] The global branch can extract global features of different scales of the child chest image. The global branch is provided with a global feature block at each level. Each global feature block can extract a global feature map of the child chest image at the current level scale, and input the obtained global feature map to the fusion branch of the next level and the current level. Specifically, when the child chest image is input into the global branch, the child chest image is first sliced into a plurality of 4*4 image blocks, and then flattened in the channel direction to be input into the linear embedding layer of the first level. After passing through the linear embedding layer, the channel number becomes twice. Then, the global feature block is input to extract the global feature map. The obtained global feature map is input to the MFF block (multi-scale feature fusion block) and the second level. In the second level and the third level and the fourth level, the feature map output from the previous level is first sliced and combined, and then the global feature block is used to extract the global feature map. The obtained feature map is output to the next level and the MFF block.

[0046] Since the child chest image has obvious intra-class variation and inter-class similarity, it is very important to obtain global semantic information. Therefore, two attention modules are provided in each global feature block. The first attention module adopts a window multi-head self-attention mechanism (W-MSA), and the second attention module adopts a shift window multi-head self-attention mechanism (SW-MSA). Compared with the multi-head self-attention (MSA) module in the Transformer, the W-MSA module maps the features into a window size of MxM, arranges the windows one after another, and performs self-attention calculation on each window, which can effectively reduce the computational complexity. The computational complexity formulas of the multi-head self-attention and the window multi-head self-attention are as follows:

[0047] Ω(MSA)=4hwC 2 +2(hw) 2 C,Ω(W-MSA)=4hwC 2 +2M 2 hwC,

[0048] wherein h represents the height of the feature map, w represents the width of the feature map, C represents the depth of the feature map, and M represents the size of each window.

[0049] The network architecture of the global feature block is as shown in the upper right corner. A residual connection with a relative position bias (rel.pos.) is also adopted after each module of the global feature block. The specific expression is as follows: Figure 1

[0050] g i =f 1×1 (W-MSA(LN(G i-1 )))+G i-1 ,G i =f 1×1 (SW-MSA(LN(g​i )))+g i ,

[0051] where G i and g i represent the output features of the SW-MSA and W-MSA of the global feature block. f 1×1 is a convolution operation with a kernel size of 1x1, which is equivalent to a linear operation. LN is a normalization (also known as layer norm) operation.

[0052] The local branch can extract local features of different scales of the child chest image. The local branch is provided with a local feature block at each level. Each local feature block can extract a local feature map of the child chest image at the current level scale, and input the obtained local feature map to the fusion branch of the next level and the current level.

[0053] When the child chest image is input into the local branch, the child chest image is first input into a two-dimensional convolution layer with a convolution kernel size of 4*4 and a stride of 4, and then input into the local feature block after normalization at the first level. The local feature map is extracted, and the extracted local feature map is input into the second level and the MFF block. In the second level and the third level and the fourth level, the input local feature map is first normalized, and then input into the local feature block after convolution operation by a two-dimensional convolution layer with a convolution kernel size of 2*2 and a stride of 2. The model architecture of the local feature block is shown in the upper right corner of Figure 1 , which includes a first convolution module and a second convolution module. The first convolution module uses a depth convolution with a size of 3*3, which can effectively reduce the FLOPs (Floating-point Operations Per Second) of the network and improve the calculation efficiency. The second convolution module uses a group convolution with a size of 1*1, where the number of groups of the group convolution is equal to the number of channels of the input feature map. Then, the layer norm and GELU activation function in the Transformer are used to perform cross-channel information interaction through a linear layer to obtain the local feature map. The specific expression is: L i =f 1×1 (LN(f d3×3 (L i-1 )))+L i-1 ,

[0054] where L i represents the output feature of the local feature block. f d3×3 is a depth convolution operation with a kernel size of 3x3, f 1×1 is a group convolution operation with a kernel size of 1*1.

[0055] In order to improve the accuracy of feature extraction of chest radiograph images of children, it is very important to fuse local features and global features from different levels. Therefore, a fusion branch is adopted to fuse the global feature maps output by the global feature block, the local feature maps output by the local feature block, and the fused feature maps output by the fusion branch of the previous level. An MFF block is arranged on each level of the fusion branch. The MFF block, also known as a multi-scale feature fusion block, can adaptively fuse the local feature maps, the global feature maps, and the fused feature maps output by the previous level according to the input features. The network architecture of the MFF block is shown in Figure 2 The expression of the channel attention mechanism is as follows: CA(x) = σ(MLP(AvgPool(x)) + MLPMaxPoolx, and σ is a Sigmoid function. Figure 2 i The expression of the channel attention mechanism is as follows: CA(x) = σ(MLP(AvgPool(x)) + MLPMaxPoolx, and σ is a Sigmoid function. i The expression of the channel attention mechanism is as follows: CA(x) = σ(MLP(AvgPool(x)) + MLPMaxPoolx, and σ is a Sigmoid function. i-1 The expression of the channel attention mechanism is as follows: CA(x) = σ(MLP(AvgPool(x)) + MLPMaxPoolx, and σ is a Sigmoid function. i The expression of the channel attention mechanism is as follows: CA(x) = σ(MLP(AvgPool(x)) + MLPMaxPoolx, and σ is a Sigmoid function.

[0056] The fusion module includes a channel attention branch and a spatial attention branch. Since the self-attention in the global feature block can capture global information in space and time to a certain extent, the MFF block feeds back the input global feature maps to the channel attention branch. The channel attention branch adopts a channel attention (CA) mechanism to extract features from the global feature maps output by the global feature block, so as to improve the feature representation of specific semantics by utilizing the mutual dependence between channel mappings. The local feature maps are fed back to the spatial attention branch, and the spatial attention branch adopts a spatial attention (SA) mechanism to extract features from the local feature maps output by the local feature block, so as to enhance local details and suppress irrelevant regions.

[0057] The expression of the channel attention mechanism is as follows: CA(x) = σ(MLP(AvgPool(x)) + MLPMaxPoolx, and σ is a Sigmoid function. Figure 2 The expression of the channel attention mechanism is as follows: CA(x) = σ(MLP(AvgPool(x)) + MLPMaxPoolx, and σ is a Sigmoid function.

[0058] The expression of the channel attention mechanism is as follows: CA(x) = σ(MLP(AvgPool(x)) + MLPMaxPoolx, and σ is a Sigmoid function.

[0059] ​The expression of the spatial attention mechanism is: SA(x) = σ(f 7×7 Concat[AvgPool(x), MaxPool(x)]), f 7×7 is a convolution operation with a convolution kernel size of 7*7.

[0060] The expression of the fusion module is:

[0061]

[0062]

[0063]

[0064]

[0065] wherein, represents element-wise multiplication, is generated by the channel attention combination, is generated by the spatial attention combination, is generated by the down-sampling of the previous level of the MFF block. is the result after the fusion of the global feature, the local feature, and the feature after the fusion of the previous level.

[0066] Finally, the fused feature output by the second fusion unit the feature generated by the channel attention combination the feature generated by the spatial attention combination are connected together and generate a feature F i In the residual inverted multi-layer perceptron module, a plurality of convolution modules are provided to perform convolution operations on the input fused feature. Referring to Figure 2 The multi-layer perceptron module of the present embodiment includes a first convolution module, a second convolution module, and a third convolution module connected in sequence, the first convolution module adopts a depth convolution with a size of 3*3, the second convolution module adopts four depth convolutions with a size of 1*1, and the third convolution module adopts a depth convolution with a size of 1*1.

[0067] The expression of the residual inverted MLP operation is:

[0068]

[0069] IRMLP(x) = f 1×1 (f 1×1 (f 3×3 (LN(x)) + LN(x)).

[0070] wherein, is generated by channel attention combination, is generated by spatial attention combination, is generated by the downsampling of the previous level of the MFF block. is the result after the fusion of the global feature, the local feature and the fused feature of the previous level.

[0071] Figure 3 The difference between the MFF block of the application and the network structures of ResNet, Swin Transformer, ConvNeXt, etc. is shown. The use of the MFF block structure can to some extent avoid the problems of gradient disappearance, explosion and network degradation, thereby effectively capturing the global and local feature information of each level.

[0072] The three-branch parallel structure used in this embodiment can maximize the preservation of the local features and global features of the child chest image without interfering with each other.

[0073] The classification module is used to obtain the probability of belonging to various child pneumonia such as viral pneumonia, bacterial pneumonia, mycoplasma pneumonia, and mixed bacterial and viral pneumonia according to the features output by the chest feature extraction module, and obtain the classification result of child pneumonia. The classification module can be various classification networks commonly used in the art, such as Figure 1 As shown in the figure, the classification module in this embodiment includes a global average pooling layer, a normalization layer and a linear embedding layer.

[0074] In order to further improve the accuracy of the classification result, in some embodiments, the classification module is a multi-modal fusion module, which can fuse the features of the child chest image output by the chest feature extraction module and the features of the blood protein information to obtain the classification result of child pneumonia. The features of the blood protein information come from the data obtained after normalizing the data in the blood protein information.

[0075] The multi-modal fusion method includes model-independent method and model-based method. The model-independent method does not depend on a specific deep learning method, and integrates the features immediately after extracting the features, which is also called early fusion. The model-based method needs to use a deep learning model to explicitly solve the multi-modal fusion problem, such as kernel-based method, image model method and neural network method, etc., and the fusion is performed after the output of the classification or regression result, which is also called late fusion. The model-independent multi-modal fusion method is used in this embodiment, and the features of the child chest image and the features of the blood protein information are fused at the feature level after being extracted and the result is output. The multi-modal fusion method is a conventional technical means for fusing models in the art, which will not be described here.

[0076] In some embodiments, a data preprocessing module is also provided for image preprocessing of the child chest X-ray images and data cleaning of the blood protein information. For example, the child chest X-ray images are subjected to preprocessing operations such as rotation, cropping, normalization, etc.; the null values of the blood protein data are removed, and the white blood cell count and C-reactive protein concentration values are retained, and the medical record number is associated with the child chest X-ray image, etc.

[0077] In training the model in the classification system, the data set used includes an X-ray chest X-ray data set and a blood protein information data set. The X-ray chest X-ray data set mainly consists of two parts, a private data set and a public data set. The private data set is the first hospital chest X-ray of children with community-acquired pneumonia (CAP) collected in a hospital from 2015 to 2018. The X-ray chest X-ray is taken in the posterior-anterior or anterior-posterior position by a dedicated radiographic device (DR7500; Kodak Healthineers). Retrospective analysis of these clinically diagnosed CAP images of children screened 300 cases of viral pneumonia, 300 cases of bacterial pneumonia and 300 cases of mycoplasma pneumonia. In addition, the public data set is from the ChestXRay2017 data set. In order to analyze the X-ray chest X-ray, first, all the X-ray chest X-rays are screened to remove all low-quality or unreadable scans to ensure the quality of the pictures, then two professional doctors grade the diagnosis of the images, and finally to reduce the image diagnosis error, a third expert also checks the test set. This data set contains a total of 5856 X-ray chest X-rays, of which 1583 are normal lung images and 4273 are bacterial and viral mixed pneumonia images. In total, the X-ray chest X-ray data set contains a total of 6756 child X-ray chest X-rays, which are divided into training set: validation set: test set = 7: 1: 2 ratio, and the X-ray chest X-ray is rotated and cropped, and augmix (a data enhancement method) is used for data enhancement to improve data quality.

[0078] The blood protein information includes white blood cell count (WBC) and C-reactive protein concentration (CRP) from blood routine test. The processing of the blood protein information data set is to clean the data, remove the null values, retain the medical record number and the WBC and CRP values of the day, and correspond one by one with the image data. The value of white blood cell count is (4-10) x 10 9 / L is the normal range, and 4 x 10 9 / L and 10 x 10 9 / L is the critical point, and the number of cases of three different types of pneumonia is counted. The difference in white blood cell count between the three different types of pneumonia is statistically significant. The normal range of C-reactive protein concentration is (0-8) mg / L, and 8 mg / L is the critical point for counting the number of cases of three different types of pneumonia. The difference in C-reactive protein concentration between the three different types of pneumonia is statistically significant.

[0079] In summary, the child pneumonia classification system of the embodiment sets multiple levels to extract feature maps of different scales, and sets global branches, local branches and fusion branches, uses a window multi-head self-attention mechanism to extract global feature maps of the child chest image on the global branches, uses convolution operation to extract local feature maps of the child chest image on the local branches, and fuses the global feature maps, the local feature maps and the fused feature maps of the previous level extracted at each level on the fusion branches, and then classifies to obtain the child pneumonia classification result. By using multi-scale hierarchical features, the intra-class variation and inter-class similarity of child chest images are solved, and the accuracy, sensitivity and specificity of the model classification result are effectively improved.

[0080] Exemplary method

[0081] As shown in Figure 4 The embodiment of the present application also provides a child chest image classification method based on hierarchical multi-scale feature fusion, and specifically, the classification method comprises the following steps:

[0082] Step S100: Extract global feature maps and local feature maps of the child chest image on multiple levels, and the scales of the feature maps extracted by all levels are different from each other, a window multi-head self-attention mechanism is used to extract the global feature maps, and convolution operation is used to extract the local feature maps;

[0083] Step S200: Fuse the global feature maps, the local feature maps and the fused feature maps of the previous level of each level to obtain the fused feature maps of each level;

[0084] Step S300: Linearly classify the fused feature maps of the last level to obtain the child chest image classification result;

[0085] It should be noted that the child chest image classification method of the embodiment can also be used to classify adult chest images.

[0086] In the embodiment, the child chest image classification method based on the hierarchical multi-scale feature fusion network can refer to the corresponding description in the child pneumonia classification system based on hierarchical multi-scale feature fusion, and will not be repeated here.

[0087] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0088] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for description, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for convenient distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0089] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0090] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or in combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different ways to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0091] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented by other ways. For example, the above-described apparatus / terminal device embodiments are only schematic, and for example, the division of the above modules or units is only a logical function division, and there can be another division way in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0092] The above integrated modules / units, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the above-mentioned computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be realized. The above-mentioned computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The above-mentioned computer readable medium can include any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the above-mentioned computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0093] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand; it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not deviate from the spirit and scope of the corresponding technical solutions, and should be included in the protection scope of the present application.

Claims

1. A child pneumonia classification system based on hierarchical multi-scale feature fusion, characterized in that, The system comprises: a chest radiograph feature extraction module and a classification module; The chest radiograph feature extraction module is configured to extract features of a child chest radiograph image, and is provided with a global branch, a local branch and a fusion branch with the same number of levels; The global branch is configured to extract global features of the child chest radiograph image at different scales, each level of the global branch is provided with a global feature block, the global feature block is configured to extract a global feature map of the child chest radiograph image based on a window multi-head self-attention mechanism, and the global feature maps output by all global feature blocks are of different scales; The local branch is configured to extract local features of the child chest radiograph image at different scales, each level of the local branch is provided with a local feature block, the local feature block is configured to extract a local feature map of the child chest radiograph image using a convolution operation, and the local feature maps and the global feature maps of the same level are of the same scale; Each level of the fusion branch is provided with a multi-scale feature fusion block, and the multi-scale feature fusion block is configured to fuse the global feature map and the local feature map of the level where the multi-scale feature fusion block is located and the fused feature map output by the multi-scale feature fusion block of the previous level; The classification module is configured to obtain a child pneumonia classification result according to the features output by the chest radiograph feature extraction module; The multi-scale feature fusion block comprises a fusion module and a residual inverted multi-layer perception module, the fusion module is configured to fuse the global feature map and the local feature map output by the global feature block and the local feature block of the level where the multi-scale feature fusion block is located and the fused feature map output by the multi-scale feature fusion block of the previous level, and input the obtained fused feature map into the multi-layer perception module; The fusion module is provided with a channel attention branch and a spatial attention branch; The fusion module is provided with a first fusion unit and a second fusion unit, the first fusion unit is configured to connect the global feature map output by the global feature block, the local feature map output by the local feature block and the fused feature map output by the multi-scale feature fusion block of the previous level, and the second fusion unit is configured to connect the features output by the channel attention branch, the features output by the spatial attention branch and the features output by the first fusion unit.

2. The hierarchical multi-scale feature fusion based children pneumonia classification system of claim 1, wherein, The global feature block is provided with a first attention module using a window multi-head self-attention mechanism and a second attention module using a shift window multi-head self-attention mechanism.

3. The hierarchical multi-scale feature fusion based children pneumonia classification system of claim 1, wherein, The local feature block is provided with a first convolution module and a second convolution module, the first convolution module uses a depth convolution with a size of 3*3, and the second convolution module uses a grouped convolution with a size of 1*1.

4. The hierarchical multi-scale feature fusion based children pneumonia classification system of claim 1, wherein, The classification module is a multi-modal fusion module, and the multi-modal fusion module is configured to multi-modal fuse the features of the child chest radiograph image and the features of blood protein information output by the chest radiograph feature extraction module to obtain a child pneumonia classification result.

5. The hierarchical multi-scale feature fusion based children pneumonia classification system of claim 1, wherein, The channel attention branch is configured to extract features of the global feature map output by the global feature block using a channel attention mechanism, and the spatial attention branch is configured to extract features of the local feature map output by the local feature block using a spatial attention mechanism.

6. The hierarchical multi-scale feature fusion based child pneumonia classification system of claim 1, wherein, The residual inverted multi-layer perception module comprises a first convolution module, a second convolution module and a third convolution module connected in sequence, the first convolution module adopts one depth convolution with a size of 3*3, the second convolution module adopts four depth convolutions with a size of 1*1, and the third convolution module adopts one depth convolution with a size of 1*1.

7. The hierarchical multi-scale feature fusion based children pneumonia classification system of claim 1, wherein, A data preprocessing module is further provided to perform image preprocessing on the child chest image and data cleaning on the blood protein information.

8. A method for classifying chest radiography images of children based on hierarchical multi-scale feature fusion, characterized in that, The method is implemented based on the child pneumonia classification system based on hierarchical multi-scale feature fusion according to any one of claims 1-7, and the method comprises: Global feature maps and local feature maps of the child chest image are extracted on multiple levels, and the scales of the feature maps extracted by all levels are different from each other, the global feature maps are extracted by using a window multi-head self-attention mechanism, and the local feature maps are extracted by using a convolution operation; The global feature maps, the local feature maps and the fused feature maps of the previous level of each level are fused to obtain fused feature maps of each level; Linear classification is performed on the fused feature maps of the last level to obtain a classification result of the child chest image.

Images