Image processing method, device, equipment and storage medium based on federated learning

Abstract

The present invention discloses an image processing method, apparatus, device and storage medium based on federated learning, the method comprising: obtaining attention features of a lung image; obtaining a pneumonia area of ​​the lung image to be tested based on the attention features and a federated lesion area detection model, the federated lesion area detection model being obtained by executing a federated learning training process based on multiple data terminals; and determining the pneumonia level of the lung image based on the regional range of the pneumonia area. The present invention can improve the efficiency and accuracy of determining or analyzing the pneumonia level.

Description

Technical Field

[0001] The present invention relates to the fields of financial technology and artificial intelligence technology, and in particular to an image processing method, device, equipment and storage medium based on federated learning. Background Art

[0002] Pneumonia is one of the most common infectious diseases in the world. When treating pneumonia, after determining the pathogen of pneumonia, it is usually necessary to first determine the grade of pneumonia, and then determine different treatment measures based on the grade of pneumonia. The existing methods for determining the grade of pneumonia mostly rely on the doctor's visual judgment. However, pneumonia cases usually grow in stages, and the number and capacity of doctors in a single hospital are limited, which will affect the efficiency and accuracy of determining the grade of pneumonia. Therefore, the means of determining the grade of pneumonia needs to be further improved. Summary of the invention

[0003] The main purpose of the present invention is to provide an image processing method, apparatus, device and storage medium based on federated learning, aiming to improve the efficiency and accuracy of determining the grade of pneumonia.

[0004] To achieve the above objectives, in a first aspect, the present invention provides an image processing method based on federated learning, comprising:

[0005] Obtaining attention features for lung images;

[0006] Based on the attention feature and the federated lesion region detection model, a pneumonia region of the lung image to be tested is obtained, wherein the federated lesion region detection model is obtained by executing a federated learning training process based on multiple data terminals;

[0007] The pneumonia grade of the lung image is determined based on the area range of the pneumonia area.

[0008] Furthermore, before acquiring the attention features of the lung image and the lesion area detection model based on federated learning, the method further includes:

[0009] Performing convolution processing on the sample lung image with the lesion area outlined, to obtain a sample attention feature corresponding to the sample lung image;

[0010] Based on the sample lung image and the sample attention feature, construct a local training sample set;

[0011] Training a local model based on the local training sample set, and sending local model parameters to a coordination terminal, wherein the coordination terminal is used to jointly process the local model parameters uploaded by multiple data terminals to obtain joint model parameters;

[0012] The joint model parameters sent by the coordination end are received to continue iterative training of the local model, and when the model converges, the federal lesion area detection model is obtained.

[0013] Optionally, the region range is represented by a first volume of the pneumonia region, and determining the pneumonia grade of the lung image based on the region range of the pneumonia region includes:

[0014] determining a first volume of the pneumonia region and a second volume of the lung region according to the lung image;

[0015] The pneumonia grade of the lung image is determined based on a threshold interval in which the ratio of the first volume to the second volume is located.

[0016] Optionally, before determining the first volume of the pneumonia area and the second volume of the lung area according to the lung image, the method further includes:

[0017] Determining the number of slices, scanning slice thickness and slice spacing of the lung image;

[0018] The determining the first volume of the pneumonia area and the second volume of the lung area according to the lung image comprises:

[0019] Determine the first volume according to the pneumonia area and the number of slices, the scanning slice thickness and the slice spacing;

[0020] The second volume is determined according to the lung region, the number of slices, the scanning slice thickness, and the slice spacing.

[0021] Optionally, determining the pneumonia grade of the lung image based on the area range of the pneumonia area includes:

[0022] Extracting the pathological features of the pneumonia area;

[0023] The pneumonia grade of the lung image is determined based on the regional range and lesion characteristics of the pneumonia area.

[0024] Optionally, the acquiring of attention features of the lung image includes:

[0025] Performing feature extraction processing on the lung image to obtain image features of the lung image;

[0026] Performing a first convolution process on the image feature to obtain a plurality of benchmark attention features, wherein the elements in the benchmark attention features are benchmark attentions corresponding to each feature point in the image feature;

[0027] Performing a second convolution process on the image features to obtain an attention coefficient matrix, wherein the elements in the attention coefficient matrix are benchmark attention coefficients corresponding to each of the feature points in the image features, and the number of channels of the attention coefficient matrix is ​​the same as the number of the benchmark attention features;

[0028] Based on the image features, each of the benchmark attention features and the attention coefficient matrix, the attention features of each of the feature points are obtained, and the attention features of the lung image are composed of the attention features of each of the feature points.

[0029] Optionally, obtaining the attention feature of each of the feature points based on the image feature, each of the benchmark attention features and the attention coefficient matrix includes:

[0030] Based on the image feature and each of the benchmark attention features, obtaining a benchmark feature vector corresponding to each benchmark attention feature, wherein the benchmark feature vector is obtained by summing the products of the image feature and corresponding elements in the benchmark attention feature;

[0031] Multiplying each of the benchmark feature vectors by the attention coefficient matrix to obtain a product matrix for each benchmark feature vector;

[0032] The product values ​​of the same feature points in the product matrix of each benchmark feature vector are added together to obtain the attention features of each feature point.

[0033] In a second aspect, the present invention further provides an image processing device based on federated learning, comprising:

[0034] An acquisition module, used to acquire attention features of lung images;

[0035] A processing module, configured to obtain the pneumonia region of the lung image to be tested based on the attention feature and a federated lesion region detection model, wherein the lesion region detection model is obtained by executing a federated learning training process based on multiple data terminals;

[0036] A determination module is used to determine the pneumonia grade of the lung image based on the area range of the pneumonia area.

[0037] Furthermore, the device also includes: a building module and a sending module.

[0038] The processing module is further used to perform convolution processing on the sample lung image with the lesion area outlined, so as to obtain the sample attention feature corresponding to the sample lung image;

[0039] The construction module is used to construct a local training sample set based on the sample lung image and the sample attention feature;

[0040] The sending module is used to train a local model based on the local training sample set and send local model parameters to the coordination end, and the coordination end is used to jointly process the local model parameters uploaded by multiple data ends to obtain joint model parameters;

[0041] The determination module is also used to receive the joint model parameters sent by the coordination end to continue iterative training of the local model, and obtain the federal lesion area detection model when the model converges.

[0042] Optionally, the region range is represented by a first volume of the pneumonia region, and the determining unit includes:

[0043] A first determining unit, configured to determine a first volume of the pneumonia region and a second volume of the lung region according to the lung image;

[0044] The second determining unit is configured to determine the pneumonia grade of the lung image based on a threshold interval in which the ratio of the first volume to the second volume is located.

[0045] Optionally, the first determination unit is specifically used to determine the number of layers, scanning layer thickness and layer spacing of the lung image;

[0046] Determine the first volume according to the pneumonia area and the number of slices, the scanning slice thickness and the slice spacing;

[0047] The second volume is determined according to the lung region, the number of slices, the scanning slice thickness, and the slice spacing.

[0048] Optionally, the determination module is specifically used to extract the pathological features of the pneumonia area;

[0049] The pneumonia grade of the lung image is determined based on the regional range and lesion characteristics of the pneumonia area.

[0050] Optionally, the acquisition module includes:

[0051] A processing unit, used for performing feature extraction processing on the lung image to obtain image features of the lung image;

[0052] The processing unit is further configured to perform a first convolution process on the image feature to obtain a plurality of benchmark attention features, wherein the elements in the benchmark attention features are benchmark attentions corresponding to each feature point in the image feature;

[0053] The processing unit is further used to perform a second convolution process on the image features to obtain an attention coefficient matrix, wherein the elements in the attention coefficient matrix are benchmark attention coefficients corresponding to each of the feature points in the image features, and the number of channels of the attention coefficient matrix is ​​the same as the number of the benchmark attention features;

[0054] An acquisition unit is used to obtain the attention features of each of the feature points based on the image features, each of the baseline attention features and the attention coefficient matrix, wherein the attention feature matrix is ​​composed of the attention features of each of the feature points.

[0055] Optionally, the acquisition unit is specifically used to obtain a reference feature vector corresponding to each reference attention feature based on the image feature and each reference attention feature, wherein the reference feature vector is obtained by summing the products of the image feature and corresponding elements in the reference attention feature;

[0056] Multiplying each of the benchmark feature vectors by the attention coefficient matrix to obtain a product matrix for each benchmark feature vector;

[0057] The product values ​​of the same feature points in the product matrix of each benchmark feature vector are added together to obtain the attention features of each feature point.

[0058] In the third aspect, the present invention also proposes an image processing device based on federated learning, wherein the lesion area detection device comprises: a memory, a processor, and a lesion area detection program stored in the memory and executable on the processor, and the image processing program based on federated learning is executed by the processor to perform the steps of the image processing method based on federated learning.

[0059] In a fourth aspect, the present invention further proposes a computer-readable storage medium, on which a federated learning-based image processing program is stored, and when the federated learning-based image processing program is executed by a processor, the steps of the federated learning-based image processing method are implemented.

[0060] The present invention provides an image processing method, device, equipment and storage medium based on federated learning. After acquiring the attention feature of the lung image, the pneumonia area of ​​the lung image to be tested can be obtained based on the attention feature and the federal lesion area detection model, and the lesion area detection model is obtained by executing the federated learning training process based on multiple data terminals; and the pneumonia level of the lung image can be automatically determined based on the regional range of the pneumonia area, so that the lung image can be qualitatively analyzed for the pneumonia level, play an auxiliary role, and improve the efficiency of determining or analyzing the pneumonia level. In addition, since the combination of attention features can reduce the influence of features other than pneumonia features, and the federal lesion area detection model can realize the joint use of multiple medical resources under the premise of protecting the privacy of patients in each hospital, the accuracy of determining or analyzing the pneumonia level can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1It is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention;

[0062] Figure 2 It is a flowchart of the first embodiment of the image processing method based on federated learning of the present invention;

[0063] Figure 3 A schematic diagram of a network structure of a lung lobe segmentation method according to an embodiment of the present invention;

[0064] Figure 4 This is a flowchart of step S10 of image processing based on federated learning in an embodiment of the present invention;

[0065] Figure 5 A flowchart showing step S14 in an image processing method based on federated learning according to an embodiment of the present invention

[0066] Figure 6 Functional schematic module diagram of a preferred embodiment of the image processing device based on federated learning of the present invention.

[0067] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0068] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0069] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0070] It should be noted that the image processing method based on federated learning provided in an embodiment of the present invention can be used to improve the efficiency and accuracy of determining the level of pneumonia. The executor of the method can be any image processing device based on federated learning. For example, the image processing method based on federated learning can be executed by a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc., without specific limitation herein.

[0071] like Figure 1As shown, the image processing device based on federated learning may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0072] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation on the image processing device based on federated learning, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0073] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an image processing program based on federated learning. The operating system is a program that manages and controls the hardware and software resources of the device, and supports the operation of the image processing program based on federated learning and other software or programs.

[0074] exist Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the federated learning-based image processing program stored in the memory 1005, and execute the federated learning-based image processing method of an embodiment of the present invention.

[0075] Based on the above structure, various embodiments of an image processing method based on federated learning are proposed.

[0076] Reference Figure 2 , Figure 2 Schematic diagram of the flow of the first embodiment of the image processing method based on federated learning of the present invention.

[0077] S10: Obtain attention features of lung images;

[0078] S20: obtaining the pneumonia region of the lung image to be tested based on the attention feature and the federated lesion region detection model, wherein the federated lesion region detection model is obtained by executing a federated learning training process based on multiple data terminals;

[0079] S30: Determine the pneumonia grade of the lung image based on the area range of the pneumonia area.

[0080] In some possible implementations, the lung image of the embodiment of the present invention can be obtained directly by shooting CT (Computed Tomography), or can also be obtained by transmission through other electronic devices. In the embodiment of the present invention, the acquired lung image can be a lung image under three viewing angles, and the three viewing angles include a sagittal plane viewing angle, a coronal plane viewing angle, and a cross-sectional viewing angle. Correspondingly, multiple tomographic images, i.e., lung images, can be obtained at each viewing angle, and the lung images at the same viewing angle can be constructed to form a three-dimensional lung image. For example, multiple layers of images at the same viewing angle can be stacked to obtain a three-dimensional lung image, or linear fitting or surface fitting can be performed to obtain a three-dimensional lung image.

[0081] In the following embodiments, the lung image may be a lung image of any layer at any viewing angle, which is not specifically limited in the embodiments of the present invention.

[0082] In addition, when performing pulmonary lobe segmentation processing on the lung image to obtain the pulmonary lobe segmentation image, the obtained pulmonary lobe segmentation image may include: at least one of the left lung image and the right lung image. The left lung image includes: the left upper lobe and the left lower lobe; the right lung image includes: the right upper lobe, the right middle lobe, and the right lower lobe, that is, the pulmonary lobe segmentation image is a pulmonary lobe image corresponding to any one or more pulmonary lobes of the five pulmonary lobes. The embodiment of the present invention can obtain the segmentation images of the five pulmonary lobes respectively through the pulmonary lobe segmentation processing.

[0083] In some possible implementations, a lung image is subjected to lung lobe segmentation processing to obtain a lung lobe segmentation image, comprising: step 1: acquiring the lung lobe fissure features in the sagittal plane, the lung lobe fissure features in the coronal plane, and the lung lobe fissure features in the transverse plane of the lung image; step 2: correcting the third lung lobe fissure feature using any two of the lung lobe fissure features in the sagittal plane, the coronal plane, and the transverse plane; and step 3: segmenting the lung image using the corrected lung lobe fissure features.

[0084] Step 1: Obtain the lobar fissure features of the lung image in the sagittal plane, the lobar fissure features in the coronal plane, and the lobar fissure features in the transverse plane.

[0085] In some possible implementations, the lobe fissure features of the lung images at different viewing angles may be extracted by feature extraction processing. The lobe fissure features are features used to perform segmentation processing of each lobe region in the lung image. For example, the lobe fissure features may be used to determine the position of the fissure surface between the lobes to achieve lobe segmentation.

[0086] The embodiment of the present invention can perform feature extraction processing on the lung images in the sagittal plane, coronal plane and cross-sectional perspectives respectively to obtain the fissure features of the lung images in the corresponding perspectives, that is, the lobe fissure features of the lung images in the sagittal plane, the lobe fissure features in the coronal plane and the lobe fissure features in the cross-section can be obtained respectively. In the embodiment of the present invention, the lobe fissure features in each perspective can be expressed in the form of a matrix or a vector, and the lobe fissure features can represent the characteristic values ​​of the lung images at each pixel point in the corresponding perspective.

[0087] In some possible implementations, feature extraction processing of fissure features of lung images can be performed by a feature extraction neural network. For example, a neural network can be trained to achieve accurate extraction of lobe fissure features of lung images by the neural network, and lobe segmentation can be performed using the obtained features. When the accuracy of lobe segmentation exceeds the accuracy threshold, it means that the accuracy of the lobe fissure features obtained by the neural network meets the requirements. At this time, the network layer that performs segmentation in the neural network can be removed, and the retained network portion can be used as a feature extraction neural network of an embodiment of the present invention. Among them, the feature extraction neural network can be a convolutional neural network, such as a residual network, a pyramid feature network, or a U network. The above is only an exemplary description and is not a specific limitation of the present invention.

[0088] Step 2: Correct the third lobe fissure feature using any two lobe fissure features in the sagittal plane, coronal plane and transverse plane.

[0089] In some possible implementations, when the lobe fissure features at three perspectives are obtained, the lobe fissure features at two perspectives can be used to correct the lobe fissure features at the third perspective to improve the accuracy of the lobe fissure features at the third perspective.

[0090] In one example, an embodiment of the present invention can use the lobe fissure features in the coronal and cross-sectional perspectives to correct the lobe fissure features in the sagittal perspective. In other embodiments, another lobe fissure feature can also be corrected by any two of the lobe fissure features in the three perspectives. For the convenience of description, the correction of the third lobe fissure feature by the first lobe fissure feature and the second lobe fissure feature is described in the following embodiment. The first lobe fissure feature, the second lobe fissure feature and the third lobe fissure feature correspond to the lobe fissure features in the three perspectives of the embodiment of the present invention, respectively.

[0091] In some possible implementations, the first lobe fissure feature and the second lobe fissure feature can be converted to the perspective of the third lobe fissure feature by mapping, and the two lobe fissure features obtained by mapping can be used to perform feature fusion with the third lobe fissure feature to obtain a corrected lobe fissure feature.

[0092] Step 3: Segment the lung image using the corrected lobe fissure features.

[0093] In some possible implementations, lobe segmentation can be performed directly through the corrected lobe fissure feature to obtain the segmentation result of the lobe fissure. Alternatively, in other implementations, the corrected lobe fissure feature and the third lobe fissure feature can also be subjected to feature fusion processing, and lobe segmentation is performed based on the fusion result to obtain the segmentation result of the lobe fissure. Among them, the segmentation result may include the position information corresponding to each partition in the identified lung image. For example, the lung image may include five lobe regions, namely the right upper lobe, the right middle lobe, the right lower lobe, the left upper lobe and the left lower lobe, and the segmentation result may include the position information of the above five lobes in the lung image. The embodiment of the present invention can represent the segmentation result by means of mask features, that is, the segmentation result obtained by the embodiment of the present invention can be a feature represented in the form of a mask. For example, the embodiment of the present invention can assign unique corresponding mask values, such as 1, 2, 3, 4 and 5, to the above five lobe regions, and the area formed by each mask value is the location area where the corresponding lobe is located. The above mask values ​​are only exemplary, and other mask values ​​can also be configured in other embodiments.

[0094] Based on the above embodiments, the features of the lung lobe fissures under three perspectives can be fully integrated to improve the information content and accuracy of the corrected fissure features, thereby improving the accuracy of the lung lobe segmentation results.

[0095] In order to explain the embodiment of the present invention in detail, each process of the embodiment of the present invention is described below respectively.

[0096] In the embodiment of the present invention, the method for obtaining the lobar fissure features of the lung image in the sagittal plane, the lobar fissure features in the coronal plane, and the lobar fissure features in the cross-section is:

[0097] A multi-sequence lung image in the sagittal plane, the coronal plane and the cross-section is obtained; and lobe fissure features are extracted from the multi-sequence lung images in the sagittal plane, the coronal plane and the cross-section, respectively, to obtain lobe fissure features in the sagittal plane, lobe fissure features in the coronal plane and lobe fissure features in the cross-section.

[0098] The embodiment of the present invention can first obtain a multi-sequence lung image at three viewing angles. As described in the above embodiment, multi-layer lung images (multi-sequence images) of the lung image at different viewing angles can be collected by CT imaging, and a three-dimensional lung image can be obtained through the multi-layer lung images at each viewing angle.

[0099] When a multi-sequence lung image under three viewing angles is obtained, feature extraction processing can be performed on each lung image, for example, by performing feature extraction processing on the lung image under each viewing angle through the above-mentioned feature extraction neural network, and obtaining the lobe fissure features of each image under the three viewing angles, such as the lobe fissure features under the sagittal plane, the lobe fissure features under the coronal plane, and the lobe fissure features under the cross-section. Among them, since each viewing angle can include multiple lung images, the embodiment of the present invention can perform feature extraction processing of the multiple lung images in parallel through multiple feature extraction neural networks, thereby improving feature extraction efficiency.

[0100] Figure 3 FIG. 4 is a schematic diagram showing a network structure of a lung lobe segmentation method according to an embodiment of the present invention. Figure 3 As shown, the network that performs feature extraction processing in the embodiment of the present invention can be a U network (U-net) or other convolutional neural networks that can perform feature extraction.

[0101] When the lobe fissure features of the lung images under various viewing angles are obtained, the lobe fissure features of any two of the sagittal plane, the coronal plane and the transverse plane may be used to correct the third lobe fissure feature. The process may include:

[0102] The arbitrary two pulmonary lobe fissure features are mapped to the viewing angle of the third pulmonary lobe fissure feature; and the third pulmonary lobe fissure feature is corrected using the mapped arbitrary two pulmonary lobe fissure features.

[0103] For the convenience of description, the correction of the third lobe fissure feature by the first lobe fissure feature and the second lobe fissure feature is taken as an example to illustrate below.

[0104] Since the extracted lobe fissure features are different under different viewing angles, the embodiment of the present invention can convert the lobe fissure feature maps under three viewing angles to one viewing angle. Among them, the method of mapping any two lobe fissure features to the viewing angle where the third lobe fissure feature is located is: mapping the lobe fissure features of any two multi-sequence lung images of the sagittal plane, coronal plane and cross-section to the viewing angle where the third lobe fissure feature is located. In other words, the first lobe fissure feature and the second lobe fissure feature can be converted to the viewing angle where the third lobe fissure feature is located. Among them, through the mapping conversion of the viewing angle, the feature information of the viewing angle before mapping can be integrated into the lobe fissure feature obtained after mapping.

[0105] As described in the above embodiment, the embodiment of the present invention can obtain multiple lung images at each viewing angle, and the multiple lung images correspond to multiple pulmonary lobe fissure features. Each feature value in the pulmonary lobe fissure feature corresponds one-to-one to each pixel point of the corresponding lung image.

[0106] The embodiment of the present invention can determine the position mapping relationship between each pixel in the lung image when the perspective is converted to another perspective based on a three-dimensional lung image formed by multiple lung images at one perspective, such as when a pixel is switched from the first position of the first perspective to the second position of the second perspective, the feature value corresponding to the first position at the first perspective is mapped to the second position. Through the above embodiment, the mapping conversion between the lobe fissure features of each lung image at different perspectives can be achieved.

[0107] In some possible implementations, when the lobe fissure features of three perspectives are mapped to the same perspective, the two mapped lobe fissure features can be used to perform correction processing on the third lobe fissure feature to improve the information content and accuracy of the third lobe fissure feature.

[0108] In the embodiment of the present invention, the method for correcting the third lobe fissure feature by using any two of the mapped lobe fissure features is:

[0109] The spatial attention feature fusion is performed using the mapped features of any two pulmonary lobe fissures and the third pulmonary lobe fissure feature to obtain a first fusion feature and a second fusion feature; and the corrected third pulmonary lobe fissure feature is obtained based on the first fusion feature and the second fusion feature.

[0110] In the embodiment of the present invention, the feature after the first lobe fissure feature is mapped may be referred to as the first mapping feature, and the feature after the second lobe fissure feature is mapped may be referred to as the second mapping feature. When the first mapping feature and the second mapping feature are obtained, a spatial attention feature fusion between the first mapping feature and the third lobe fissure feature may be performed to obtain a first fused feature, and a spatial attention feature fusion between the second mapping feature and the third lobe fissure feature may be performed to obtain a second fused feature.

[0111] The method of fusing the spatial attention features by respectively using the mapped features of any two pulmonary lobe fissures and the third pulmonary lobe fissure to obtain the first fused feature and the second fused feature is:

[0112] Respectively connect any two of the lobe fissure features with the third lobe fissure feature to obtain a first connection feature and a second connection feature; perform a first convolution operation on the first connection feature to obtain a first convolution feature, and perform a first convolution operation on the second connection feature to obtain a second convolution feature; perform a second convolution operation on the first convolution feature to obtain a first attention coefficient, and perform a second convolution operation on the second convolution feature to obtain a second attention coefficient; use the first convolution feature and the first attention coefficient to obtain the first fusion feature, and use the second convolution feature and the second attention coefficient to obtain the second fusion feature.

[0113] In some possible implementations, such as Figure 3 As shown, the above-mentioned spatial attention feature fusion processing can be performed by the network module of the spatial attention mechanism. The embodiment of the present invention takes into account the different importance of lobe fissure features at different positions and adopts a spatial attention mechanism. Among them, the convolution processing based on the attention mechanism can be implemented through the spatial attention neural network (attention), and the important features are further highlighted in the obtained fusion features. In the training process of the spatial attention neural network, the importance of each position of the spatial feature can be adaptively learned to form an attention coefficient with the feature object at each position. For example, the coefficient can represent a coefficient value in the interval [0,1]. The larger the coefficient, the more important the feature at the corresponding position.

[0114] In the process of performing spatial attention fusion processing, the first mapping feature and the third lobe fissure feature can first be concatenated to obtain a first connection feature, and the second mapping feature and the third lobe fissure feature can be concatenated to obtain a second connection feature. The above connection processing can be concatenated in the channel direction. In an embodiment of the present invention, the scales of the first mapping feature, the second mapping feature, and the third lobe fissure feature can all be identified as (C / 2, H, W), where C represents the number of channels of each feature, H represents the height of the feature, and W represents the width of the feature. Correspondingly, the scales of the first connection feature and the second connection feature obtained by the connection processing can be expressed as (C, H, W).

[0115] When the first connection feature and the second connection feature are obtained, the first convolution operation can be performed on each of the first connection feature and the second connection feature, such as using the convolution layer A to perform the first convolution operation through a 3*3 convolution kernel, and then batch normalization (bn) and activation function (relu) processing can be performed to obtain the first convolution feature corresponding to the first connection feature and the second convolution feature corresponding to the second connection feature. The scale of the first convolution feature and the second convolution feature can be expressed as (C / 2, H, W). The first convolution operation can reduce the parameters in the feature map and reduce the subsequent calculation cost.

[0116] In some possible implementations, when the first convolution feature and the second convolution feature are obtained, the second convolution operation and the sigmoid function processing can be performed on the first convolution feature and the second convolution feature respectively to obtain the corresponding first attention coefficient and second attention coefficient respectively. The first attention coefficient can represent the importance of the features of each element of the first convolution feature, and the second attention coefficient can represent the importance of the features of the elements in the second convolution feature.

[0117] like Figure 3 As shown, for the first convolution feature or the second convolution feature, two convolution layers B and C can be used to perform the second convolution operation, wherein the convolution layer B is processed by a 1*1 convolution kernel, and then batch normalization (bn) and activation function (relu) are performed to obtain the first intermediate feature. The scale of the first intermediate feature map can be expressed as (C / 8, H, W), and then the second convolution layer C is used to perform a convolution operation of a 1*1 convolution kernel on the first intermediate feature map to obtain a second intermediate feature map of (1, H, W). Further, the second intermediate feature map can be processed by an activation function using a sigmoid function to obtain the attention coefficient corresponding to the first convolution feature or the second performance feature, and the coefficient value of the attention coefficient can be a value in the range of [0,1].

[0118] Through the above-mentioned second convolution operation, the first connection feature and the second connection feature can be subjected to dimensionality reduction processing to obtain the attention coefficient of a single channel.

[0119] In some possible implementations, when a first attention coefficient corresponding to a first convolution feature and a second attention coefficient corresponding to a second convolution feature are obtained, a product process may be performed on the first convolution feature and the first attention coefficient, and the product result may be added to the first convolution feature to obtain a first fused feature. And a product process may be performed on the second convolution feature and the second attention coefficient matrix, and the product result may be added to the second convolution feature to obtain a second fused feature. Among them, the product process (mul) may be corresponding element multiplication, and the feature addition (add) may be corresponding element addition. In the above manner, effective fusion of features from three perspectives may be achieved.

[0120] Alternatively, in some other implementations, the feature obtained by multiplying the first convolution feature by the first attention coefficient may be added to the first convolution feature, and the added feature may be subjected to several convolution operations to obtain the first fused feature; and the feature obtained by multiplying the second convolution feature by the second attention coefficient may be added to the second convolution feature, and the added feature may be subjected to several convolution operations to obtain the second fused feature. In this way, the accuracy of the fused feature may be further improved, and the information content of the fusion may be improved.

[0121] When the first fusion feature and the second fusion feature are obtained, the first fusion feature and the second fusion feature may be used to obtain a corrected third lobe fissure feature.

[0122] In some possible implementations, since the first fused feature and the second fused feature respectively include feature information from three perspectives, the first fused feature and the second fused feature can be directly connected, and a third convolution operation can be performed on the connected features to obtain a corrected third lobe fissure feature. Alternatively, the first fused feature, the second fused feature, and the third lobe fissure feature can be connected, and a third convolution operation can be performed on the connected features to obtain a corrected third lobe fissure feature.

[0123] The third convolution operation may include group convolution processing. The third convolution operation may further achieve further fusion of feature information in each feature. Figure 3 As shown, the third convolution operation of the embodiment of the present invention may include grouped convolution D (depth wise conv), wherein the grouped convolution can speed up the convolution speed while improving the accuracy of the convolution features.

[0124] When the corrected third lobe fissure feature is obtained through the third convolution operation, the corrected lobe fissure feature can be used to segment the lung image. In the embodiment of the present invention, the segmentation result corresponding to the corrected lobe fissure feature can be obtained by convolution. Figure 3 As shown, in the embodiment of the present invention, the corrected lobe fissure features can be input into the convolution layer E, and the standard convolution is performed through the 1*1 convolution kernel to obtain the segmentation result of the lung image. As described in the above embodiment, the segmentation result can represent the location areas where the five lobes in the lung image are located. Figure 3 As shown, each lung lobe area in the lung image is distinguished by light and dark filling colors.

[0125] Based on the above configuration, the lung lobe segmentation method based on multiple perspectives provided by the embodiment of the present invention can solve the technical problem of not fully utilizing the information from other perspectives to segment the lung lobes, resulting in information loss and inability to accurately segment the lung lobes.

[0126] As described in the above embodiments, the embodiments of the present invention can be implemented by a neural network, such as Figure 3 As shown, the neural network for executing the lung lobe segmentation method under multi-viewpoints in an embodiment of the present invention may include a feature extraction neural network, a spatial attention neural network, and a segmentation network (including convolutional layers D and E).

[0127] The embodiment of the present invention may include three feature extraction neural networks, which are respectively used to extract the features of the lobe fissures under different viewing angles. Among them, the three feature extraction networks can be referred to as the first branch network, the second branch network and the third branch network. Among them, the structures of the three branch networks in the embodiment of the present invention are exactly the same, and the input images of each branch network are different. For example, the sagittal lung image samples are input to the first branch network, the coronal lung image samples are input to the second branch network, and the transverse lung image samples are input to the third branch network, so as to respectively perform feature extraction processing of the lung image samples under each viewing angle.

[0128] Specifically, in the embodiment of the present invention, the process of training the feature extraction neural network includes:

[0129] Acquire training samples in the sagittal plane, the coronal plane and the transverse plane, wherein the training samples are lung image samples with marked lobe fissure features; use the first branch network to perform feature extraction on the lung image samples in the sagittal plane to obtain a first predicted lobe fissure feature; use the second branch network to perform feature extraction on the lung image samples in the coronal plane to obtain a second predicted lobe fissure feature; use the third branch network to perform feature extraction on the lung image samples in the transverse plane to obtain a third predicted lobe fissure feature; use the first predicted lobe fissure feature, the second predicted lobe fissure feature and the third predicted lobe fissure feature and the corresponding marked lobe fissure feature to obtain the network losses of the first branch network, the second branch network and the third branch network, and use the network losses to adjust the parameters of the first branch network, the second branch network and the third branch network.

[0130] As described in the above embodiment, the first branch network, the second branch network and the third branch network are respectively used to perform feature extraction processing of lung image samples in the sagittal plane, the coronal plane and the cross-sectional perspective, and then the corresponding predicted features, namely the first predicted lobe fissure feature, the second predicted lobe fissure feature and the third predicted lobe fissure feature, can be obtained.

[0131] When each predicted lobe fissure feature is obtained, the first predicted lobe fissure feature, the second predicted lobe fissure feature, and the third predicted lobe fissure feature can be used together with the corresponding labeled lobe fissure feature to obtain the network losses of the first branch network, the second branch network, and the third branch network. For example, the loss function of the embodiment of the present invention can be a logarithmic loss function, and the network loss of the first branch network can be obtained by the first predicted lobe fissure feature and the labeled real lobe fissure feature, the network loss of the second branch network can be obtained by the second predicted lobe fissure feature and the labeled real lobe fissure feature, and the network loss of the third branch network can be obtained by the third predicted lobe fissure feature and the labeled real lobe fissure feature.

[0132] When the network loss of each branch network is obtained, the parameters of the first branch network, the second branch network and the third branch network can be adjusted according to the network loss of each network until the termination condition is met. Among them, the embodiment of the present invention can use the network loss of any branch of the first branch network, the second branch network and the third branch network to adjust the network parameters of the first branch network, the second branch network and the third branch network, such as convolution parameters, etc., respectively. Thereby, the network parameters under any perspective can be correlated with the features under the other two perspectives, the correlation between the extracted lobe fissure features and the lobe fissure features under the other two perspectives can be improved, and the preliminary fusion of the lobe fissure features under each perspective can be achieved.

[0133] In addition, the training termination condition of the feature extraction neural network is that the network loss of each branch network is less than the first loss threshold, which indicates that each branch network of the feature extraction neural network can accurately extract the lobe fissure features of the lung image under the corresponding perspective.

[0134] When the feature extraction neural network is trained, the feature extraction neural network, the spatial attention neural network and the segmentation network can be used for training at the same time, and the segmentation result output by the segmentation network and the corresponding marking result in the marked lobe fissure feature can be used to determine the network loss of the entire neural network. The network loss of the entire neural network is further used to feedback and adjust the network parameters of the feature extraction neural network, the spatial attention neural network and the segmentation network until the network loss of the entire neural network is less than the second loss threshold. In the embodiment of the present invention, the first loss threshold is greater than or equal to the second loss threshold, so that the network accuracy of the network can be improved.

[0135] When applying the neural network of the embodiment of the present invention to perform lung lobe segmentation based on multiple perspectives, the lung images of the same lung under different perspectives can be input into the three branch networks respectively, and finally the final lung image segmentation result can be obtained through this neural network.

[0136] To summarize, the lung lobe segmentation method provided in the embodiment of the present invention can fuse multi-view feature information and perform lung lobe segmentation of lung images, thereby solving the problem of not fully utilizing information from other viewpoints to segment the lung lobes, resulting in information loss and inability to accurately segment the lung lobes.

[0137] When a segmented lung lobe image corresponding to a lung image is obtained, a lesion prediction can be performed on the segmented lung lobe image, that is, whether a preset lesion exists in the segmented lung lobe image. The type of lesion in the embodiment of the present invention can be set according to demand, for example, it can be at least one of a lung nodule and lung cancer, or it can be other types, and the present invention does not specifically limit this. It should be noted that the segmented lung lobe image can be an image corresponding to any lung lobe of the total lung image, and the embodiment of the present invention can detect the lesion area of ​​any lung lobe.

[0138] In the case of obtaining a lung image, S10, the specific method of obtaining the attention feature of the lung image can be: first obtain the image feature of the lung image, and in the case of obtaining the image feature of the lung image, the image feature can be used to obtain the attention feature matrix of the lung image. Among them, the image feature can be obtained by performing feature extraction processing on the lung image. Among them, the convolution operation of the lung image can be performed through a residual network, a pyramid feature extraction network, or other convolutional neural network to obtain the image feature of the lung image. The image feature is represented in the form of a matrix or a vector. And the image feature of the lung image can be convolved to obtain the attention feature of the lung image.

[0139] Figure 4 A flowchart of step S10 of image processing based on federated learning according to an embodiment of the present invention is shown. In which, the image features of the lung image are subjected to convolution processing to obtain an attention feature matrix, including:

[0140] S11: performing feature extraction processing on the lung image to obtain image features of the lung image;

[0141] S12: performing a first convolution process on the image features of the lung image to obtain a plurality of benchmark attention features, wherein the elements in the benchmark attention features are benchmark attentions corresponding to each feature point in the image features;

[0142] S13: performing a second convolution process on the image features to obtain an attention coefficient matrix, wherein the elements in the attention coefficient matrix are benchmark attention coefficients corresponding to each of the feature points in the image features, and the number of channels of the attention coefficient matrix is ​​the same as the number of the benchmark attention features;

[0143] S14: Based on the image features, each of the benchmark attention features and the attention coefficient matrix, the attention features of each of the feature points are obtained, and the attention features of the lung image are composed of the attention features of each of the feature points.

[0144] In the embodiment of the present invention, steps S12 and S13 may be performed simultaneously or separately, and the order of the two steps is not specifically limited.

[0145] Through step S12, the first convolution processing can be performed on the image features to obtain multiple benchmark attention features, wherein the first convolution kernel can be used to perform convolution processing on the image features in the feature direction to obtain a preset number of benchmark attention features. The preset number can be a preset value, such as 16, 32, etc., which can be set by the technicians in this field according to the needs. The elements in the obtained benchmark attention features are the benchmark attentions corresponding to each feature point in the image features. That is, the benchmark attention for each feature point can be obtained through the first convolution processing, and the benchmark attention can be in the form of a feature vector. In addition, the length of each benchmark attention feature obtained is the same as the length of the image feature, and the width of the benchmark attention feature is the same as the width of the image feature.

[0146] Among them, the feature points described in the embodiment of the present invention can be pixel points of the lung lobe segmentation image, and the first convolution kernel can be a 1*1 convolution kernel, but it is not a specific limitation of the present invention.

[0147] In step S13, a plurality of benchmark attention features can be obtained through the first convolution process, each benchmark attention feature can include benchmark attention corresponding to each feature point in the image feature, and each benchmark attention feature obtained is different. Moreover, the length of the benchmark attention feature is the same as the length of the corresponding feature point in the image feature, and the width in the benchmark attention feature is the same as the width of the corresponding feature point in the image feature, that is, the benchmark attention corresponding to the feature point of the benchmark attention feature corresponds one-to-one to the feature value of each feature point in the image feature. For example, for an image feature with a dimension of a×b×c, the dimension of each benchmark attention feature is a×b×1.

[0148] In addition, through step S13, a second convolution process can be performed on the image features to obtain an attention coefficient matrix, wherein the second convolution kernel can be used to perform convolution process on the image features to obtain an attention coefficient matrix, and the elements in the attention coefficient matrix are the benchmark attention coefficients corresponding to each feature point of the image feature. Moreover, the number of channels of the attention coefficients corresponding to each feature point in the benchmark attention coefficient matrix is ​​the same as the number of benchmark attention features. That is, each element in the benchmark attention coefficient matrix can represent multiple benchmark attention coefficients corresponding to the corresponding feature point, and the number of the benchmark attention coefficients is the same as the number of benchmark attention features. That is, each benchmark attention for the feature point can correspond to the corresponding benchmark attention coefficient.

[0149] The second convolution kernel may be a 1*1 convolution kernel, which is not specifically limited in the embodiment of the present invention. The first convolution kernel and the second convolution kernel in the embodiment of the present invention may be different convolution kernels.

[0150] In step S13, in the attention coefficient matrix obtained by performing the second convolution process on the image feature, the elements in each attention coefficient matrix are multiple attention coefficients for different feature points. That is, through the second convolution process, an attention coefficient matrix representing the attention coefficients of each feature point can be obtained, and each attention coefficient matrix can include attention coefficients corresponding to each feature point in the image feature. And the length of the obtained attention coefficient matrix is ​​the same as the length of the image feature, the width of the attention coefficient matrix is ​​the same as the width of the image feature, and the obtained attention coefficient matrix can be expressed as a×b×n, where n is the number of reference attention features.

[0151] After obtaining the attention coefficient matrix and the baseline attention feature, step S14 can be performed based on the two types of feature matrices obtained.

[0152] Among them, in an embodiment of the present invention, the obtaining of a benchmark feature vector corresponding to each benchmark attention feature based on the image feature and each benchmark attention feature includes: multiplying the feature vector corresponding to each feature point in the image feature with the first attention of the same feature point in the benchmark attention feature to obtain a first product for each feature point in the benchmark attention feature; adding the first products for each feature point in the benchmark attention feature to obtain a benchmark feature vector corresponding to the benchmark attention feature.

[0153] Among them, after obtaining the benchmark attention feature, the benchmark attention feature can be multiplied by the vector value corresponding to the corresponding feature point in the image feature to obtain the first product of the feature point for the benchmark attention feature. For example, image feature A can be expressed as [a1, b1, c1], a benchmark attention feature B can be expressed as [a2, b2, c2], and another benchmark attention feature C can be expressed as [a3, b3, c3]. Then step S23 can obtain the first product of the three feature points for the benchmark attention feature B, namely a1*a2, b1*b2 and c1*c2, and the first product of the three feature points for the benchmark attention feature C, namely a1*a3, b1*b3 and c1*c3. Similarly, the first product value of each feature point for each benchmark attention feature can be obtained.

[0154] Furthermore, the first product for each feature point in the baseline attention feature can be summed to obtain a baseline feature vector corresponding to the baseline attention feature. That is, after obtaining the first product value of each identical feature point in a baseline attention feature, the first product of each feature point can be summed to obtain a baseline feature vector. For example, for baseline attention feature B, the sum vector a1*a2, b1*b2 and c1*c2 can be obtained, that is, the baseline feature vector a1*a2+b1*b2+c1*c2 for baseline attention feature B can be obtained. Similarly, the baseline feature vector a1*a3+b1*b3+c1*c3 for baseline attention feature C can be obtained. By analogy, the baseline feature vector of each baseline attention feature can be obtained.

[0155] In the above configuration of the embodiment of the present invention, it is not necessary to calculate the correlation between each feature point and other feature points and construct a correlation tensor about each feature point, which greatly improves the calculation speed and reduces the consumption of calculation resources.

[0156] In addition, after obtaining the attention coefficient matrix, the attention features of each feature point can be obtained through step S23. Figure 5 A flowchart of step S14 in an image processing method based on federated learning according to an embodiment of the present invention is shown, wherein the obtaining of the attention features of each of the feature points based on the image features, each of the benchmark attention features and the attention coefficient matrix may include:

[0157] S141: Based on the image feature and each of the benchmark attention features, obtaining a benchmark feature vector corresponding to each benchmark attention feature, wherein the benchmark feature vector is obtained by summing the products of the image feature and corresponding elements in the benchmark attention feature;

[0158] S142: multiplying each of the reference feature vectors by the attention coefficient matrix to obtain a product matrix for each reference feature vector;

[0159] S143: Add the product values ​​of the same feature point in the product matrix of each attention coefficient matrix to obtain the attention feature of the feature point.

[0160] After obtaining the baseline feature vector corresponding to each baseline attention feature, the baseline feature vector can be multiplied with the attention coefficient matrix obtained in step S13 to obtain a product matrix corresponding to the baseline feature vector. Among them, the baseline attention and attention coefficient corresponding to the same feature point can be multiplied. Since each element in the attention coefficient matrix can include multiple attention coefficients of a feature point, and the number of attention coefficients of each feature point is the same as the number of baseline attention features, a one-to-one product of the attention coefficient and the baseline attention feature can be obtained. Therefore, after the product, multiple product values ​​of the attention coefficient of each feature point and the baseline attention can be obtained. Therefore, the product matrix corresponding to each baseline attention vector can be obtained through the above, and the product matrix includes the above-mentioned multiple product values ​​of each feature point.

[0161] After obtaining the product matrix of each benchmark attention vector, the product values ​​of the corresponding feature points in each product matrix can be added to obtain the attention features for the feature points. As described above, each product matrix can include the product value of the attention coefficient corresponding to each feature point and the benchmark attention. Therefore, the product values ​​of the corresponding attention coefficients of the same point in each product matrix can be added to obtain the attention features for the feature point, thereby obtaining the attention features of each feature point. The attention features obtained from each feature point can constitute an attention feature matrix. According to the above embodiment, the attention of each feature point in the lung lobe segmentation image can be obtained. In the above processing of the embodiment of the present invention, the attention of each feature point is obtained by multiplying the obtained attention coefficient matrix with the benchmark attention vector, and performing a simple addition operation. This process reduces the amount of calculation compared to the calculation process in the prior art.

[0162] Step S10 of the embodiment of the present invention can be implemented by a neural network with an attention mechanism, and the neural network can be a network structure that has been trained to meet the accuracy requirements, and can include a convolution module that performs the first convolution kernel and the second convolution processing, and an operation module that performs the multiplication and addition processing. When achieving global relevance (i.e., determining the attention of each feature point), resource consumption can be reduced, and the task of learning global relevance using the attention module can be completed with significantly fewer resources; and global relevance can be learned on significantly larger inputs.

[0163] When the attention features of each feature point of the lung image are obtained, the attention features can be used to perform pneumonia area detection. Among them, the embodiment of the present invention can directly obtain the pneumonia area through feature processing of the attention feature matrix formed by each feature point, or the attention matrix can be optimized before executing step S30. For example, the optimization process may include inputting the image features and the attention feature matrix composed of the attention features corresponding to each feature point into the residual module for summing processing to obtain an optimized attention feature matrix, and the elements in the optimized attention feature matrix are the optimized attention for each feature point. That is, the attention feature matrix and the image features corresponding to the original input data can be summed to obtain an optimized attention matrix.

[0164] The feature information in the attention feature matrix obtained in the embodiment of the present invention highlights the characteristics of the pneumonia area, so when performing pneumonia area prediction through the attention feature matrix, the accuracy of the prediction can be improved. Similarly, performing level prediction through an optimized attention matrix can further integrate the characteristics of the lung image, thereby further improving the accuracy of detection and level prediction of the pneumonia area. As described in the above embodiments, the present invention realizes that the pneumonia level prediction of the lung image can be performed using the attention feature matrix or the optimized attention feature matrix. The following description takes the attention feature matrix as an example, and the process of performing level prediction through the optimized attention matrix will not be repeated.

[0165] Specifically, in an embodiment of the present invention, S20, the specific method of obtaining the pneumonia area of ​​the lung image to be tested based on the attention feature and the federal lesion area detection model can be: directly inputting the attention feature into the federal lesion area detection model, and the pneumonia area of ​​the lung image to be tested can be detected by the federal lesion area detection model. The federal lesion area detection model is obtained by executing a federated learning training process based on multiple data terminals; because the combination of attention features can reduce the influence of features other than pneumonia features, and the federal lesion area detection model can realize the joint use of multiple medical resources under the premise of protecting the privacy of patients in each hospital, thereby improving the accuracy of determining or analyzing the level of pneumonia.

[0166] In an embodiment of the present invention, before S10, the process of pre-training to obtain a federal lesion area detection model may be: performing convolution processing on a sample lung image with a lesion area outlined to obtain a sample attention feature corresponding to the sample lung image; constructing a local training sample set based on the sample lung image and the sample attention feature; training a local model based on the local training sample set, and sending the local model parameters to a coordination end, the coordination end being used to jointly process the local model parameters uploaded by multiple data ends to obtain joint model parameters; receiving the joint model parameters sent by the coordination end to continue iterative training of the local model, and when the model converges, obtaining the federal lesion area detection model.

[0167] Among them, the specific process of performing convolution processing on the sample lung image with the lesion area outlined to obtain the sample attention feature corresponding to the sample lung image is the same as S10, and the embodiment of the present invention is not limited here. The coordination end can be a terminal or a server end, and the multiple data ends can be terminals set up in various medical institutions, such as terminals set up in disease centers, hospitals, clinics, and physical examination centers. The server end can be a central trusted end, and each data end sends encrypted model parameters by distributing public keys or private keys to ensure the security of model parameters or user transmission. Specifically, the encryption algorithm can be a homomorphic encryption algorithm. The coordination end is used to jointly process the local model parameters uploaded by multiple data ends to obtain the joint model parameters. The local model parameters uploaded by multiple data ends can be weighted averaged, and then the model parameters after weighted average processing are used as joint model parameters. After obtaining the joint model parameters, the coordination end can detect whether the model to be trained set at the coordination end is in a convergence state. If it is detected that it is in a non-convergence state, the joint model parameters are sent to each data end to continue iterative training of the local model. If it is determined to be in a convergence state, the joint model parameters are sent to each data end as the final model parameters and each data end is notified that the training is completed, so that each data end obtains the federal lesion area detection model according to the final model parameters.

[0168] It should be noted that the coordination end can determine whether the model is in a convergence state based on the loss value, training duration, or number of trainings. The loss value can be the difference between the latest joint model parameter and the last joint model parameter. If the model is in a convergence state based on the loss value, when the loss value is less than a preset loss threshold, such as the preset loss threshold is 0.001, or 0.00001, the model is determined to be in a convergence state, otherwise, the model is determined to be in a convergence state. If the model is in a convergence state based on the number of trainings, when the number of trainings is less than a preset number threshold, such as the preset number threshold is 1000 times, or 10000 times, the model is determined to be in a convergence state, otherwise, the model is determined to be in a convergence state.

[0169] For the embodiment of the present invention, the regional range is represented by the first volume of the pneumonia area S30, and the determining the pneumonia level of the lung image based on the regional range of the pneumonia area includes: determining the first volume of the pneumonia area and the second volume of the lung area according to the lung image; and determining the pneumonia level of the lung image based on a threshold interval in which a ratio of the first volume to the second volume is located.

[0170] Among them, lesions accumulated in the alveoli or interstitial lungs are collectively referred to as pneumonia; lesions accumulated in the bronchi are called bronchopneumonia. The classification method of pneumonia levels can be divided according to actual conditions, such as pneumonia levels can be divided into mild levels, moderate levels, or severe levels, can be divided into early levels, progressive levels, severe levels, and can also be divided into early levels, progressive levels, severe levels, and recovery levels. Different levels correspond to different threshold intervals. For example, the threshold interval corresponding to the early level is [0-0.4.5], the threshold interval corresponding to the progressive stage is [0.4.5-0.8], and the threshold interval corresponding to the severe level is [0.8-1]. When the calculated ratio is 0.7, it is determined that the pneumonia level is a progressive level. The second volume of the lung area can be the total volume of the lung area, that is, the pneumonia level of the lung image is determined by the proportion of the pneumonia area to the lung area.

[0171] It should be noted that determining the first volume of the pneumonia area and the second volume of the lung area based on the lung image includes: determining the number of layers, scanning layer thickness and inter-layer spacing of the lung image; determining the first volume based on the pneumonia area and the number of layers, scanning layer thickness and inter-layer spacing; determining the second volume based on the lung area and the number of layers, scanning layer thickness and inter-layer spacing.

[0172] In an embodiment of the present invention, the method for determining the first volume based on the pneumonia area, the number of layers, the scanning layer thickness and the layer spacing includes: gridding the pneumonia area to obtain a first gridded area of ​​the pneumonia area; and obtaining the first volume of the pneumonia area based on the first gridded area, the number of layers, the scanning layer thickness and the layer spacing.

[0173] In an embodiment of the present invention, a method for determining the second volume based on the lung region and the number of layers, the scanning layer thickness and the inter-layer spacing includes: gridding the lung region to obtain a second gridded area of ​​the lung region; and obtaining the second volume of the lung region based on the second gridded area, the number of layers, the scanning layer thickness and the inter-layer spacing.

[0174] Specifically, the meshing of the pneumonia area to obtain the first meshing area of ​​the pneumonia area includes: determining the edge of the pneumonia area; drawing a mesh in the pneumonia area, refining the mesh when the mesh touches the edge, stopping the refinement after the number of meshes reaches the set number, and calculating the sum of all mesh areas of the pneumonia area to obtain the first mesh area of ​​the pneumonia area. More specifically, the edge of each layer of the pneumonia area can be determined respectively; a first standard shape can be determined in each layer of the pneumonia area respectively; step 2: the first standard shape extends to the edge of each layer of the pneumonia area; step 3: when several set points of the standard shape touch the edge of each layer of the pneumonia area, the first standard shape stops extending and generates a second standard shape outside the standard shape; generate in accordance with steps 2 and 3 in sequence until the number of standard shapes reaches the set number, calculate the area of ​​all standard shapes, and obtain the first mesh area of ​​each layer of the pneumonia area; wherein the areas of the standard shape, the second standard shape, and the standard shape corresponding to the set number decrease in sequence.

[0175] Specifically, the meshing of the lung region to obtain the second meshing area of ​​the pneumonia region includes: determining the edge of the lung region; meshing in the pneumonia region, when the mesh touches the edge, refining the mesh, stopping the refining when the number of meshes reaches the set number, and calculating the sum of all mesh areas of the lung region to obtain the second mesh area of ​​the lung region. More specifically, the edge of each layer of the lung region can be determined respectively; a first standard shape can be determined in each layer of the lung region respectively; step 2: the first standard shape extends to the edge of each layer of the lung region; step 3: when several set points of the standard shape touch the edge of each layer of the lung region, the first standard shape stops extending and generates a second standard shape outside the standard shape; generate in accordance with steps 2 and 3 in sequence until the number of standard shapes reaches the set number, calculate the area of ​​all standard shapes, and obtain the second mesh area of ​​each layer of the lung region; wherein the areas of the standard shape, the second standard shape, and the standard shape corresponding to the set number decrease in sequence.

[0176] In an embodiment of the present invention, the first standard shape is at the geometric center of the emphysema region or the small airway lesion region. The standard shape is any one or more of a circle, an ellipse, a rectangle, and a square.

[0177] In an embodiment of the present invention, the specific process of determining the first volume according to the pneumonia area and the number of layers, the scanning layer thickness and the inter-layer spacing may include: using the first grid area corresponding to each adjacent two layers of pneumonia, as well as the inter-layer spacing and the layer thickness, to obtain the sub-volume formed by the adjacent two layers of pneumonia, and using the sum of the sub-volumes formed by all adjacent two layers of pneumonia to obtain the first volume of the pneumonia area. Among them, the adjacent two layers of structures that meet the formation of pneumonia can be regarded as prisms, and the areas of the upper and lower bottom surfaces of the prisms are the grid areas corresponding to pneumonia. The height of the prism can be determined by the inter-layer spacing and the layer thickness. For example, if the number of layers is N, the height of the prism formed by the first and second layers of emphysema can be the sum of the layer thickness X2 and the inter-layer spacing, and the height of the prism formed by the remaining two adjacent layers of pneumonia can be the sum of the layer thickness and the inter-layer spacing. The sub-volume of pneumonia formed by the adjacent layers can be determined based on the upper and lower bottom areas and the height. Then the second volume of pneumonia can be obtained by the sum of the sub-volumes. That is, since pneumonia may exist in multiple places, the first volume of the pneumonia area is the total volume of all emphysema.

[0178] In an embodiment of the present invention, the second volume is determined according to the lung region and the number of layers, the scanning layer thickness and the inter-layer spacing, including: using the second grid area corresponding to each lung region, the inter-layer spacing and the layer thickness, to obtain the sub-volume formed by the adjacent lung region, and using the sum of the sub-volumes formed by all adjacent two layers of lung regions to obtain the second volume of the lung region. Among them, the adjacent two layers of structures that satisfy the lung region can be regarded as a prism, and the area of ​​the upper and lower bottom surfaces of the prism is the grid area corresponding to the lung region. The height of the prism can be determined by the inter-layer spacing and the layer thickness. For example, if the number of layers is N, the height of the prism formed by the first and second layers of the lung region can be the sum of the layer thickness*2 and the inter-layer spacing, and the height of the prism formed by the remaining two adjacent layers of the lung region can be the sum of the layer thickness and the inter-layer spacing. The sub-volume of the lung region formed by the adjacent layers can be determined based on the upper and lower bottom areas and the height. Then, the second volume of the lung region can be obtained by the sum of the sub-volumes. That is, since there may be multiple lung regions, the second volume of the lung region is the total volume of all lung regions.

[0179] In an embodiment of the present invention, determining the pneumonia level of the lung image based on the regional range of the pneumonia area includes: extracting lesion features of the pneumonia area; and determining the pneumonia level of the lung image based on the regional range and lesion features of the pneumonia area.

[0180] Among them, different pathogens cause different types of pneumonia, among which, the pathogens that cause patients to suffer from pneumonia may include viruses, bacteria, chlamydia, mycoplasma, etc. Pneumonia types can be divided into bronchopneumonia, lobar pneumonia, spherical pneumonia, and interstitial pneumonia. Therefore, the type of pneumonia can be determined based on the test data, and the type of pneumonia can also be determined based on the lung image and the test data. Specifically, the lung image can be feature extracted by a feature extraction model pre-trained based on a neural network to obtain a first feature vector, the test data can be screened to obtain a second feature vector, and then the joint feature vector of the first feature vector and the second feature vector is calculated through the fully connected layer of the neural network, and then the joint feature vector is input into a classification model pre-trained based on the neural network, and according to the classification result, the type of pneumonia corresponding to the lung image is determined, and the classification model is trained by the feature vectors of confirmed cases and the type of pneumonia.

[0181] In addition, the lesion characteristics and lesion range of pneumonia are different at different stages or at different levels. For example, for COVID-19, the early lesion characteristics are: local presentation, mainly patchy, subsegmental or segmental distribution, subpleural distribution, ground glass density with or without thickening of interlobular septa; the lesion characteristics in the progressive stage increase, the range expands, and multiple lobes are involved; some lesions become dense, GGO coexists with consolidation or linear shadows; a small amount of pleural effusion occurs in a few cases; the lesion characteristics in the severe stage are: diffuse lesions in both lungs, a few present "white lung" manifestations; mainly consolidation shadows, combined with ground glass density, often accompanied by linear shadows; air bronchograms. The lesion characteristics in the recovery stage are: the disappearance of pulmonary gas caused by pulmonary atelectasis and alveolar effusion, substantial changes in lung tissue, a large amount of exudate filling the alveolar cavity, and the reduction or disappearance of gas in the alveolar cavity. Therefore, after determining that the pneumonia corresponding to the lung image is COVID-19, it is possible to determine whether the COVID-19 in the lung image is in the early, progressive or severe stage based on the regional range and lesion characteristics of the pneumonia area. By combining the regional range and lesion characteristics, the accuracy of pneumonia grade determination can be further improved.

[0182] In addition, the embodiment of the present invention also proposes an image processing device based on federated learning, referring to Figure 6 , the image processing device based on federated learning includes:

[0183] An acquisition module 10, used to acquire attention features of a lung image;

[0184] A processing module 20 is used to obtain the pneumonia area of ​​the lung image to be tested based on the attention feature and a federal lesion area detection model, wherein the lesion area detection model is obtained by executing a federated learning training process based on multiple data terminals;

[0185] The determination module 30 is used to extract the pathological features of the pneumonia area; and determine the pneumonia grade of the lung image based on the area range of the pneumonia area.

[0186] Furthermore, the device also includes: a building module and a sending module.

[0187] The processing module 20 is further used to perform convolution processing on the sample lung image with the lesion area outlined, so as to obtain a sample attention feature corresponding to the sample lung image;

[0188] The construction module is used to construct a local training sample set based on the sample lung image and the sample attention feature;

[0189] The sending module is used to train a local model based on the local training sample set and send local model parameters to the coordination end, and the coordination end is used to jointly process the local model parameters uploaded by multiple data ends to obtain joint model parameters;

[0190] The determination module 30 is further used to receive the joint model parameters sent by the coordination end to continue iterative training of the local model, and obtain the federal lesion area detection model when the model converges.

[0191] Optionally, the region range is represented by a first volume of the pneumonia region, and the determining unit includes:

[0192] A first determining unit, configured to determine a first volume of the pneumonia region and a second volume of the lung region according to the lung image;

[0193] The second determining unit is configured to determine the pneumonia grade of the lung image based on a threshold interval in which the ratio of the first volume to the second volume is located.

[0194] Optionally, the first determination unit is specifically used to determine the number of layers, scanning layer thickness and layer spacing of the lung image;

[0195] Determine the first volume according to the pneumonia area and the number of slices, the scanning slice thickness and the slice spacing;

[0196] The second volume is determined according to the lung region, the number of slices, the scanning slice thickness, and the slice spacing.

[0197] Optionally, the determination module 30 is specifically configured to determine the pneumonia grade of the lung image based on the regional range and lesion characteristics of the pneumonia area.

[0198] Optionally, the acquisition module 10 includes:

[0199] A processing unit, used for performing feature extraction processing on the lung image to obtain image features of the lung image;

[0200] The processing unit is further configured to perform a first convolution process on the image feature to obtain a plurality of benchmark attention features, wherein the elements in the benchmark attention features are benchmark attentions corresponding to each feature point in the image feature;

[0201] The processing unit is further used to perform a second convolution process on the image features to obtain an attention coefficient matrix, wherein the elements in the attention coefficient matrix are benchmark attention coefficients corresponding to each of the feature points in the image features, and the number of channels of the attention coefficient matrix is ​​the same as the number of the benchmark attention features;

[0202] An acquisition unit is used to obtain the attention features of each of the feature points based on the image features, each of the baseline attention features and the attention coefficient matrix, wherein the attention feature matrix is ​​composed of the attention features of each of the feature points.

[0203] Optionally, the acquisition unit is specifically used to obtain a reference feature vector corresponding to each reference attention feature based on the image feature and each reference attention feature, wherein the reference feature vector is obtained by summing the products of the image feature and corresponding elements in the reference attention feature;

[0204] Multiplying each of the benchmark feature vectors by the attention coefficient matrix to obtain a product matrix for each benchmark feature vector;

[0205] The product values ​​of the same feature points in the product matrix of each benchmark feature vector are added together to obtain the attention features of each feature point.

[0206] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which a federated learning-based image processing program is stored. When the federated learning-based image processing program is executed by a processor, the steps of the federated learning-based image processing method described below are implemented.

[0207] The various embodiments of the image processing device and computer-readable storage medium based on federated learning of the present invention can all refer to the various embodiments of the image processing method based on federated learning of the present invention, and will not be repeated here.

[0208] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0209] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0210] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0211] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image processing method based on federated learning, characterized in that: include: Obtaining attention features for lung images; Based on the attention feature and the federated lesion area detection model, a pneumonia area of ​​the lung image to be tested is obtained, wherein the federated lesion area detection model is obtained by executing a federated learning training process based on multiple data terminals; Determining the pneumonia grade of the lung image based on the regional range of the pneumonia area; The method further comprises: Performing pulmonary lobe segmentation processing on a lung image to obtain a pulmonary lobe segmentation image includes: acquiring pulmonary lobe fissure features in a sagittal plane, pulmonary lobe fissure features in a coronal plane, and pulmonary lobe fissure features in a transverse plane of the lung image; correcting a third pulmonary lobe fissure feature using any two pulmonary lobe fissure features in the sagittal plane, the coronal plane, and the transverse plane; and segmenting the lung image using the corrected pulmonary lobe fissure features; Lesion prediction for lung lobe segmentation images; Among them, the first lobe fissure feature and the second lobe fissure feature are converted to the perspective of the third lobe fissure feature by mapping, and the two lobe fissure features obtained by mapping are used to perform feature fusion with the third lobe fissure feature to obtain the corrected lobe fissure feature.

2. The method according to claim 1, characterized in that Before acquiring the attention features of the lung image and the lesion area detection model based on federated learning, the method further includes: Performing convolution processing on the sample lung image with the lesion area outlined, to obtain a sample attention feature corresponding to the sample lung image; Based on the sample lung image and the sample attention feature, construct a local training sample set; Training a local model based on the local training sample set, and sending local model parameters to a coordination terminal, wherein the coordination terminal is used to jointly process the local model parameters uploaded by multiple data terminals to obtain joint model parameters; The joint model parameters sent by the coordination end are received to continue iterative training of the local model, and when the model converges, the federal lesion area detection model is obtained.

3. The method according to claim 1, characterized in that The region range is represented by a first volume of the pneumonia region, and determining the pneumonia grade of the lung image based on the region range of the pneumonia region includes: determining a first volume of the pneumonia region and a second volume of the lung region according to the lung image; The pneumonia grade of the lung image is determined based on a threshold interval in which the ratio of the first volume to the second volume is located.

4. The method according to claim 3, characterized in that The determining the first volume of the pneumonia area and the second volume of the lung area according to the lung image comprises: Determining the number of slices, scanning slice thickness and slice spacing of the lung image; Determine the first volume according to the pneumonia area and the number of slices, the scanning slice thickness and the slice spacing; The second volume is determined according to the lung region, the number of slices, the scanning slice thickness, and the slice spacing.

5. The method according to any one of claims 1 to 4, characterized in that: The step of determining the pneumonia grade of the lung image based on the area range of the pneumonia area includes: Extracting the pathological features of the pneumonia area; The pneumonia grade of the lung image is determined based on the regional range and lesion characteristics of the pneumonia area.

6. The method according to any one of claims 1 to 4, characterized in that: The acquiring of the attention features of the lung image comprises: Performing feature extraction processing on the lung image to obtain image features of the lung image; Performing a first convolution process on the image feature to obtain a plurality of benchmark attention features, wherein the elements in the benchmark attention features are benchmark attentions corresponding to each feature point in the image feature; Performing a second convolution process on the image features to obtain an attention coefficient matrix, wherein the elements in the attention coefficient matrix are benchmark attention coefficients corresponding to each of the feature points in the image features, and the number of channels of the attention coefficient matrix is ​​the same as the number of the benchmark attention features; Based on the image features, each of the benchmark attention features and the attention coefficient matrix, the attention features of each of the feature points are obtained, and the attention features of the lung image are composed of the attention features of each of the feature points.

7. The method according to claim 6, characterized in that The obtaining of the attention feature of each feature point based on the image feature, each of the benchmark attention features and the attention coefficient matrix comprises: Based on the image feature and each of the benchmark attention features, obtaining a benchmark feature vector corresponding to each benchmark attention feature, wherein the benchmark feature vector is obtained by summing the products of the image feature and corresponding elements in the benchmark attention feature; Multiplying each of the benchmark feature vectors by the attention coefficient matrix to obtain a product matrix for each benchmark feature vector; The product values ​​of the same feature points in the product matrix of each benchmark feature vector are added together to obtain the attention features of each feature point.

8. An image processing device based on federated learning, characterized in that: The image processing device based on federated learning implements the image processing method based on federated learning as claimed in claim 1, comprising: An acquisition module, used to acquire attention features of lung images; A processing module, configured to obtain the pneumonia region of the lung image to be tested based on the attention feature and a federated lesion region detection model, wherein the lesion region detection model is obtained by executing a federated learning training process based on multiple data terminals; A determination module is used to determine the pneumonia grade of the lung image based on the area range of the pneumonia area.

9. An image processing device based on federated learning, characterized in that: The federated learning-based image processing device comprises: a memory, a processor, and a federated learning-based image processing program stored in the memory and executable on the processor. When the federated learning-based image processing program is executed by the processor, the steps of the federated learning-based image processing method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an image processing program based on federated learning, and when the image processing program based on federated learning is executed by a processor, the steps of the image processing method based on federated learning as described in any one of claims 1 to 7 are implemented.

Images