A covid-19 lesion prediction system based on smart contract and self-attention

Abstract

The application discloses a COVID-19 lesion prediction system based on contract sharing and self-attention. The system includes a medical record encryption module, a lesion segmentation module based on an attention mechanism, a lesion prediction module, a combined scoring module, and a permission control module. The lesion segmentation and prediction module is composed of a convolution operation unit, an attention calculation unit, a pooling unit, a pixel fusion unit, and a classification submodule. In order to realize the high-precision segmentation task of the image, the system adopts a U-Net deep learning network architecture based on SwinT block. SwinT block is a multi-scale window interaction attention mechanism, which divides image features into different windows and uses self-attention mechanism to interact between these windows, thus better modeling different scale features in the image. This architecture combines the features of U-Net and Swin Transformer, so that more accurate results can be obtained when performing image segmentation and prediction, improving the accuracy of the system in predicting COVID-19 lesions.

Description

TECHNICAL FIELD

[0001] The application relates to the application of smart contracts in the field of cloud storage and image processing, including image segmentation and target detection technology in cloud storage and image processing, and in particular to a COVID-19 lesion prediction system based on smart contracts and self-attention. BACKGROUND

[0002] The novel coronavirus (COVID-19) has a long incubation period, multiple transmission routes, strong infectivity, and has developed multiple variants. Timely screening of suspected cases and diagnosing suspected cases is a crucial task. In this regard, lung CT (Computed Tomography) images are one of the important bases for COVID-19 screening. However, in clinical practice, lung CT images have certain limitations:

[0003] Firstly, although CT images are helpful for early screening of suspected cases, COVID-19 has similarities in imaging performance with other viral pneumonia, organized pneumonia and eosinophilic pneumonia, which can easily cause the situation of "different diseases with the same image", and only relying on the naked eye observation of imaging doctors to identify can easily lead to misdiagnosis or missed diagnosis.

[0004] Secondly, COVID-19 has a fast development process, and patients need to undergo multiple chest CT examinations to monitor the development of the disease and evaluate the treatment effect, and it is difficult for doctors to quickly compare and identify whether there is a lesion change only by naked eye observation.

[0005] In addition, data-driven artificial intelligence (AI) methods have made breakthroughs in the field of medical imaging. However, when conducting clinical verification in complex and open real scenes, there are still problems such as low image quality, limited samples, data repetition, "gold standard" diagnosis uncertainty and data set source bias, which limit the application of AI models in actual clinical application. SUMMARY

[0006] Based on the above idea, the application provides a COVID-19 lesion prediction system based on smart contracts and self-attention, which realizes accurate screening of early COVID-19, and on this basis, utilizes a data platform and a perfect medical information database sharing system, and utilizes the security of the system platform to make medical resources truly serve the public.

[0007] To achieve the above purpose, the following technical solutions are adopted:

[0008] A COVID-19 lesion prediction system based on smart contracts and self-attention includes a medical record encryption module, a lesion segmentation module based on an attention mechanism, a lesion prediction module based on an attention mechanism, a combination scoring module and a permission control module:

[0009] Medical record encryption module: the functions of smart contract Access Control Contract (ACC), smart contract JCContract (JC) and smart contract DC Contract (DC) are introduced into the cloud environment, ACC is configured according to specific requirements and access strategies, and only authenticated and authorized visitors can access encrypted CT images, so as to provide safe and reliable electronic medical record access services for visitors;

[0010] Access Control Contract (ACC): this smart contract is written according to the storage information of visitors and related electronic medical records in the cloud; ACC is responsible for verifying the legality of access requests and controlling the access rights to electronic medical records according to the access strategy;

[0011] JC Contract: this smart contract is used to record and manage the behavior and reputation information of visitors; JC maintains the blocking time of visitors, handles the violation behavior of visitors, and returns the blocking time of visitors;

[0012] DC Contract: this smart contract is a contract for delegated control; DC maintains the relationship between the principal and the delegate, and records the host address and state information of the principal;

[0013] The lesion segmentation module based on attention mechanism includes:

[0014] SwinT block operation unit: the input image enters the neural network improved based on U-net network, and the convolution self-attention layer of each layer is calculated using Swin Trasnformer;

[0015] Multi-scale fusion unit: during the pooling process of the input image, the large resolution features that are constantly operated by self-attention are reshaped into token sequences with the same size and channel as the small resolution features, and after the token sequences are connected together, subsequent image segmentation and the next stage of lesion prediction module are performed;

[0016] Mask difference unit: MDM first filters out the segmentation mask of the lesion image to be registered, and then subtracts the obtained mask pixel by pixel; finally, MDM passes the result of subtraction through a sigmoid function to produce a probability between 0 and 1 as the segmentation network result score Mask Score;

[0017] The lesion segmentation module includes a neural network based on U-net, which replaces the original convolution operation with SwinT block, and inputs the CT image of the patient into the neural network module for down-sampling feature extraction and up-sampling feature learning; a multi-scale fusion unit (MSF) is designed at the terminal of down-sampling, which models long-term dependence through self-attention mechanism to learn global context features, and restores the large resolution features into token sequences consistent with the size and channel of the small resolution features, and then models long-term dependence through self-attention mechanism to learn global context features; at the same time, a mask difference unit (MDM) is used to process the segmentation result pixel by pixel to obtain a mask score result of the segmentation network;

[0018] The lesion prediction module based on attention mechanism: the pooling result of the encoder in the segmentation network enters the prediction network to perform lesion prediction task; the information aggregated after the pooling operation is fully utilized to learn the dependency relationship between channels by using attention mechanism;

[0019] The lesion prediction module includes:

[0020] The spatial self-attention unit of static context SCSSM: taking the multi-scale fusion features in the segmentation network as input, the self-attention mechanism is used to integrate the information between multiple slices; the self-attention mechanism is used to dynamically weight the multi-scale fusion features of each two-dimensional slice after learning; by suppressing irrelevant slices, SCSSM can focus on and predict the most relevant features of COVID-19;

[0021] Dynamic adaptive self-attention unit: better capture the context spatial relationship between lesions CT in the same period, and the output result of the self-attention unit will pass through a sigmoid function to produce a probability between 0 and 1 as the prediction score of the prediction network;

[0022] Combination scoring module: linearly combines the mask score result of COVID-19 segmentation after MDM processing and the prediction score of the prediction network after training;

[0023] The permission control module saves the prediction result and controls the access permission of the prediction result through the control contract.

[0024] The COVID-19 lesion prediction system based on smart contract and self-attention, the lesion segmentation module based on attention mechanism takes U-net network as the feature extraction basis, introduces residual connection and attention mechanism, and uses SwinT block to replace the original convolution operation, so the feature extraction process of the network is as shown in the formula:

[0025] S=F Encoder +F Decoder

[0026] P i =C swin (P i-1 ),i=1,2,3,4

[0027] T i =Concat(C cbr (T i+1 ),P j ),i=1,2,3,4,j≥iA

[0028] In the above formula, F Encoder and F Decoder represent the down-sampling process and the up-sampling process of the segmentation network, respectively, which learns the long-term dependency relationship between the detailed features and the global context information of the image by performing global pooling and self-attention operation on the input image; Conv and Concat represent convolution layer and concatenation operation, respectively; C swin includes pooling operation and Swin block, and each pooling operation reduces the image pixels by half and increases the channel number by one; C cbr includes normalization operation and Relu activation function; P i and T i represent different stages to which the encoder and the decoder belong, respectively.

[0029] The spatial self-attention unit of the static context: assuming that the input feature map is S, the feature maps S 1 , S 2 and S 3 are calculated by convolution, and then their dimensions are reshaped to R n×c , where n = h x w. Next, we multiply S 1 with the element-wise horizontal transpose of S 2 to obtain the weight map P. Similarly, we perform the same operation on the remaining feature maps to obtain the corresponding self-attention maps; the formal description is as shown in the formula:

[0030] P=F scale (W 1 Conv(S),W 2 Conv(S))

[0031] In the above formula, F scale (·) is an element-wise multiplication operation, and the weighted feature map has the same size as the original feature map. W represents different weights for calculating convolution feature maps, respectively; then the weighted sum operation is performed on multiple self-attention maps, and the sigmoid activation function is applied to obtain the final spatial self-attention map P ∈ R n1×n1, where F sum represents the weighted sum:

[0032] S = W a3 Conv(S a ) + W q1 Conv(S q ) + W z3 Conv(S z )

[0033] F = F scale (Sigmod(F sum (P a , P q , P z )), S)

[0034] The dynamic adaptive self-attention unit first reshapes the feature size of the pooled fusion features of a single picture to d x 1 (d = c x h x w), and then calculates the attention weight of d x T to store the dynamic correlation between all period lesion features, so the features are connected to the feature M e R d , and the formal description is as shown in the formula:

[0035] Q ij = SoftMax(F scale (W a M, Transpose(W b M)))

[0036] M SA = F scale (Q ij , W c M)

[0037] In the above formula, Softmax(·) represents a normalized exponential function, F scale (·) is an element-wise multiplication operation, and the weighted feature map has the same size as the original feature map; W A , W B , and W C represent the weights of feature layers M A , M B , and M C , respectively; the feature mapping is input into an extended convolutional layer, which uses batch normalization and ReLU to expand the receptive field. The self-attention feature map obtained is convolved by an adaptive temporal convolution kernel ATCK to dynamically aggregate the feature information of the lesion. The formal description is as shown in the formula, where represents the ReLU activation function:

[0038]

[0039] M OUT = F scale (M SA , M ATCK );

[0040] The output result M out of the prediction network will pass through a sigmoid function to produce a probability between 0 and 1, which is the prediction score obtained by the prediction network.

[0041] The permission control in the permission control module includes:

[0042] The patient calls the DC smart contract to grant the delegation right of the electronic medical record to the delegatee, and at the same time, the patient uses his own private key to decrypt the electronic medical record to obtain the plaintext content of the electronic medical record; only the authorized delegatee and the patient himself can view the judgment result of the system;

[0043] The patient encrypts the electronic medical record using a temporary key and digitally signs the encrypted electronic medical record using his own private key; the delegatee receives the encrypted information, decrypts the electronic medical record using his own private key, and verifies the integrity of the electronic medical record using the public key of the patient; finally, the delegatee re-encrypts the decrypted electronic medical record using his own public key and uploads it to the cloud storage;

[0044] When medical institution B needs to access the electronic medical record of the patient, medical institution B calls the corresponding smart contract for access control; the smart contract verifies the identity of medical institution B, the serial number of the electronic medical record, and the legality of the access behavior; if the verification is passed, medical institution B obtains the access control permission;

[0045] At the same time, the delegatee and medical institution B generate a temporary key; the delegatee decrypts the encrypted electronic medical record using the temporary key and digitally signs the decrypted electronic medical record using his own private key; medical institution B decrypts the electronic medical record using the public key of the delegatee and verifies the integrity of the electronic medical record using the public key of the patient.

[0046] When the electronic medical record is not delegated, the patient can directly share the electronic medical record with medical institution B without the participation of the delegatee.

[0047] 1. To establish a safe and efficient artificial intelligence image assisted diagnosis system, realize the accurate screening of early COVID-19, and on this basis, standardize the data processing and improve the data quality. At the same time, we will use data platform and big data processing technology to develop and perfect the medical information database sharing system, so as to ensure the safety of the platform and make the medical resources truly serve the public. Due to the wide spread of COVID-19 virus, the traditional offline deployment mode is inefficient and has limited coverage. Therefore, the public online service platform combined with the basic software infrastructure becomes an effective way to solve this problem.

[0048] 2. By using computer vision and artificial intelligence technology, our project aims to quickly capture early lesions, accurately distinguish COVID-19 from other lung diseases, and timely share data and related treatment information on the cloud platform. By using computer vision and artificial intelligence technology, we can carry out technical transformation on the existing traditional medical industry, improve the speed and accuracy of COVID-19 diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 It is the overall flowchart of the COVID-19 lesion prediction system based on smart contract and self-attention of the present application.

[0050] Figure 2 It is the data encryption flowchart of the COVID-19 lesion prediction system based on smart contract and self-attention of the present application.

[0051] Figure 3 It is the lesion segmentation module diagram based on U-net improvement in the COVID-19 lesion prediction system based on smart contract and self-attention of the present application.

[0052] Figure 4 It is the pooling and melting flowchart in the COVID-19 lesion prediction system based on smart contract and self-attention of the present application.

[0053] Figure 5 It is the spatial self-attention module flowchart of static context in the COVID-19 lesion prediction system based on smart contract and self-attention of the present application.

[0054] Figure 6 It is the dynamic adaptive self-attention module flowchart of the COVID-19 lesion prediction system based on smart contract and self-attention of the present application.

[0055] Figure 7 It is the permission control diagram of the COVID-19 lesion prediction system based on smart contract and self-attention of the present application.

[0056] Figure 8Fig. 1 shows the labeled images used for training the COVID-19 lesion prediction system based on smart contract and self-attention; a CT picture of a COVID-19 infected patient; b a normal CT picture.

[0057] Figure 9 Fig. 4 shows the comparison curve of the experimental results of the COVID-19 course staging prediction system based on attention mechanism and two other methods. DETAILED DESCRIPTION

[0058] The present application will be described in detail below in conjunction with specific embodiments.

[0059] A COVID-19 lesion prediction system based on smart contract and self-attention, the overall process is as shown in Figure 1 , including the following:

[0060] A1 stores the CT images of patients based on smart contract and encrypts them;

[0061] The workflow of the present scheme is shown in Figure 2 . Before deploying these smart contracts, patients need to generate their own public key, private key and signature using the elliptic curve cryptography system ECC, and save these identity verification information. Then the patient starts to deploy the smart contract, introduces the functions of smart contract Access Control Contract (ACC), JC Contract (JC) and DC Contract (DC) into the cloud environment, and provides secure and reliable electronic medical record access service for visitors. In this way, ACC can be configured according to specific needs and access strategies, and only authenticated and authorized visitors can access encrypted electronic medical records.

[0062] The specific steps of step A1 are as follows:

[0063] A11. Encrypt and store electronic medical records

[0064] As shown in Figure 2 , the patient generates an electronic medical record in medical institution A, and signs the electronic medical record using the patient's private key. Then, the patient encrypts the electronic medical record using the public key provided by medical institution A to obtain an encrypted electronic medical record file. The encrypted electronic medical record is uploaded to the cloud storage by the hospital.

[0065] A12 patient creates and deploys smart contract

[0066] In step A12, the patient needs to create and deploy the following smart contracts:

[0067] Access Control Contract(ACC): This smart contract is written based on the information of visitors and the storage of relevant electronic medical records in the cloud. ACC is responsible for verifying the legality of access requests and controlling access to electronic medical records according to access policies.

[0068] JC Contract: This smart contract is used to record and manage the behavior and reputation information of visitors. JC maintains the blocking time of visitors, handles the violation behavior of visitors, and returns the blocking time of visitors.

[0069] DC Contract: This smart contract is a contract for delegation control. DC maintains the relationship between the principal and the delegate, and records the host address and state information of the principal.

[0070] A2 segmentation network to perform lesion segmentation task;

[0071] Step A2 fuses attention mechanism and residual connection as Figure 3 shown, the network is improved based on U-net network. Due to the inherent induction bias problem in convolution mechanism, existing deep learning methods are prone to lose key features of lesions. In order to alleviate this problem, we modify the encoder and replace the original convolution operation with SwinT block. At the same time, in the segmentation network, the feature information output by the encoder at each stage is usually different. If we only use convolution operation, we will not only be affected by the background area information, but also cannot capture the long-term dependence between features and global context information.

[0072] To this end, we innovatively propose a multi-scale fusion module (MSF). This module models long-term dependencies through self-attention mechanism to learn global context features. As Figure 4 shown, in the MSF module, the large resolution features are now reshaped into token sequences consistent with the size and channels of the small resolution features. Then, the token sequences of large and small resolution features are connected together for self-attention. Through transformer learning, the MSF module can more effectively solve the problem of multi-scale feature fusion, achieving higher segmentation performance.

[0073] After the segmentation network, we propose a mask difference module (MDM). Process the segmentation result pixel by pixel to get a mask score result of the segmentation network.

[0074] The specific steps of step A2 are as follows:

[0075] A21. The HDCT image stored in the electronic medical record enters the segmentation network to perform lesion segmentation task

[0076] AsFigure 3 As shown, the network structure can be divided into two parts: an encoder Encoder and a decoder Decoder. The Encoder contains 4 consecutive Encoder blocks, each of which is composed of a pooling layer MaxPool, a SwinT module and a linear embedding layer Linear Embedding, for extracting features of the input image; the Decoder is a mirror inversion of the Encoder, each Decoder block is composed of an up-sampling operation transpose convolution, a skip connection and two convolution layers, for mapping low-level features in the encoder to the segmentation result of the input image. At the same time, each convolution layer adds Batch Normalization and activation function ReLU to enhance the expression ability of the network. The skip connection connects the feature map of the corresponding layer in the encoder with the feature map of the decoder, so as to pass the low-level features to the decoder, which helps to preserve the detail information.

[0077] We take an input image sequence of 64*64*1 as an example, which is continuously down-sampled by 2 times through the encoder Encoder, to obtain the feature extraction information of the image. After feature fusion in the MSF module, it is used as the input of the decoder Decoder, and its formal description is as shown in the formula:

[0078] S=F Encoder +F Decoder

[0079] In the above formula, F Encoder and F Decoder represent the down-sampling process and the up-sampling process of the segmentation network respectively, through global pooling and self-attention operation on the input image, the long-term dependence relationship between the detail features and the global context information of the image is learned. The formal description of the feature fusion process of the network is as shown in the formula:

[0080] P i =C swin (P i-1 ),i=1,2,3,4

[0081] T i =Concat(C cbr (T i+1 ),P j ),i=1,2,3,4,j≥iA

[0082] In the above formula, Conv and Concat represent convolution layer and concatenation operation respectively; C swin includes pooling operation and Swin block, each pooling operation reduces the image pixels by half and increases the channel number by one time; C cbrincluding normalization operation and Relu activation function; P i and T i denote different stages that the encoder and decoder belong to, respectively.

[0083] A22. The feature map after segmentation network enters the mask difference unit (MDM)

[0084] As shown in Figure 2 , the MDM first filters out the segmentation mask of the lesion image to be registered, and then the MDM subtracts the obtained mask pixel by pixel. Finally, the MDM passes the result of subtraction through a sigmoid function to generate a probability between 0 and 1, which is the score mask obtained by the segmentation network.

[0085] A3. The pooling result of the encoder in the segmentation network enters the prediction network to perform the lesion prediction task;

[0086] To fully utilize the aggregated information after the pooling operation, an additional operation is needed to capture the inter-channel dependency. To achieve this goal, the operation in this series needs to meet two conditions: one is that it can learn the non-exclusive relationship and nonlinear relationship between channels, and the other is that it can prevent too many fully connected layers from being introduced, which leads to complex modeling and difficult training. To meet these conditions, the invention designs a double nonlinear feature mapping convolution to meet the first condition, and uses a dynamic adaptive self-attention module (DASM) to meet the second condition.

[0087] The specific steps of step A3 are as follows:

[0088] A31. The down-sampling result of the encoder enters the spatial self-attention unit of the static context SCSSM

[0089] Step A31 proposes a SCSSM to integrate the spatial dependency information of multiple images. The SCSSM takes the multi-scale fusion features in the segmentation network as input, and uses the self-attention mechanism to integrate the information between multiple slices. It is worth noting that the SCSSM uses the self-attention mechanism to dynamically weight the learned multi-scale fusion features of each two-dimensional slice. By suppressing irrelevant slices to encode spatial information, SCSSM can focus on and predict the most relevant features of COVID-19.

[0090] As shown in Figure 5 , the SCSSM is dynamically weighted, integrating a series of two-dimensional slice features of a patient at a single point. These blocks [S a ...S q ...S z ] are multi-scale fusion feature maps of multiple two-dimensional slices. In this patent, we choose n=3. Then the feature mapping is input into an extended convolution layer to extend the receiver domain respectively.

[0091] For the first feature map S a , we get the convolution feature maps S a1 , S a2 and S a3 , and then we reshape their dimensions to R n1×c1 , where n1 = h1 x w1. Next, we multiply S a1 with the horizontal transpose of the elements of S a2 , and get the weight map P a . Similarly, we do the same operation for the remaining feature maps, and get the corresponding self-attention maps. It is worth noting that we perform different operations on the feature map S q , and get a single convolution feature S q1 from it. This single convolution feature S q1 is multiplied by self-attention, which is obtained by the weight map after softmax. Its formal description is shown in the formula:

[0092] P a = F scale (W a1 Conv(S a ), W a2 Conv(S a ))

[0093] P q = F scale (W q1 Conv(S q ), W q1 Conv(S q ))

[0094] P z = F scale (W z1 Conv(S z ), W z2 Conv(S z ))

[0095] where F scale (·) is an element-wise multiplication operation, and the weighted feature map has the same size as the original feature map. W represents different weights for calculating convolution feature maps, respectively.

[0096] Then, we perform weighted sum operation on multiple self-attention maps, and apply sigmoid activation function to get the final spatial self-attention map P ∈ R n1×n1 . Then, we multiply the spatial self-attention map P with S, and reshape the result to R c×h×w, to get the final output feature map F. The feature map F contains multiple feature maps with rich spatial dependencies, which are described as follows: F = F sum represents the weighted sum:

[0097] S = W a3 Conv(S a )+W q1 Conv(S q )+W z3 Conv(S z )

[0098] F = F scale (Sigmod(F sum (P a ,P q ,P z )),S)

[0099] A32. Convolution operation on self-attention feature map using dynamic convolution kernel (ATCK) (dynamic adaptive self-attention unit)

[0100] As shown in Figure 6 , in order to better capture the context spatial relationship between the lesions CT of the same period, we first reshape the feature size of the pooling fusion features of a single picture to d x 1 (d = c x h x w), and then calculate the attention weight of d x T to store the dynamic correlation between all period lesion features, so the features are connected to the feature M ∈ R d .

[0101] First, we obtain M A , M B , M C by convolution feature map feature map M. Next, we use M A times the transpose of M B , and then process it through the softmax activation function to obtain the attention weight map embedding self-attention information Q i j ∈ R d*d . Then, by element-wise multiplication of M c and self-attention information, we obtain the self-attention information of the input features. The formal description is as follows:

[0102] Q ij = SoftMax(F scale (W a M, Transpose(W b M)))

[0103] M SA = F scale (Q ij ,Wc M)

[0104] where Softmax(·) denotes the normalized exponential function; F scale (·) is an element-wise multiplication operation, the weighted feature map has the same size as the original feature map; W A , W B , W C represent the weights of the feature layers M A , M B , M C , respectively.

[0105] Secondly, we input the feature map into an extended convolutional layer, using batch normalization and ReLU to expand the receptive field. We perform convolution operation on the obtained self-attention feature map through adaptive temporal convolution kernel ATCK, dynamically aggregating the feature information of the lesion. Then we perform element-wise addition operation on M out ∈R d*1 . The formal description is shown in the formula, where denotes the ReLU activation function:

[0106]

[0107] M OUT = F scale (M SA , M ATCK )

[0108] The output result M out of the prediction network will pass through a sigmoid function to produce a probability between 0 and 1, which is the prediction score obtained by the prediction network.

[0109] A4. Linearly combine the COVID-19 segmentation MDM processed mask score and the prediction score of the prediction network, with an initial ratio of 1:1, and change the score ratio weight in the continuous training process. The final result of the training is used for linear combination in the actual test to obtain the final COVID-19 infection probability score. The training ratio of this patent is 0.684:0.316;

[0110] A5. Save the prediction result and control the access permission of the prediction result through the control contract;

[0111] A51. Control access permission through smart contract

[0112] The patient calls the DC smart contract to grant the delegation right of the electronic medical record to the delegate, while the patient uses his own private key to decrypt the electronic medical record to obtain the plaintext content of the electronic medical record. Only the authorized delegate and the patient himself can view the judgment result of the system.

[0113] A52. The delegate and the patient generate a temporary key

[0114] The patient encrypts the electronic medical record using the temporary key and digitally signs the encrypted electronic medical record using his own private key. After receiving the encrypted information, the delegate decrypts the electronic medical record using his own private key and verifies the integrity of the electronic medical record using the public key of the patient. Finally, the delegate re-encrypts the decrypted electronic medical record using his own public key and uploads it to the cloud storage.

[0115] A53. Access to access rights by other medical institutions or personnel

[0116] When medical institution B needs to access the patient's electronic medical record, medical institution B calls the corresponding smart contract for access control. The smart contract verifies the identity of medical institution B, the serial number of the electronic medical record, and the legality of the access behavior. If the verification is passed, medical institution B obtains access control rights.

[0117] At the same time, the delegate and medical institution B generate a temporary key. The delegate decrypts the encrypted electronic medical record using the temporary key and digitally signs the decrypted electronic medical record using his own private key. Medical institution B decrypts the electronic medical record using the public key of the delegate and verifies the integrity of the electronic medical record using the public key of the patient.

[0118] When there is no delegation of the electronic medical record, the patient can directly share the electronic medical record with medical institution B without the participation of the delegate.

[0119] As Figure 7 shown, through the above process, the present scheme realizes the control and delegation of the electronic medical record by the patient, ensuring the security and integrity of the data. Medical institution B can obtain the patient's electronic medical record information by verification and decryption under the condition of complying with the access rights. The participation of the delegate and the generation of the temporary key increase the security, ensuring the secure transmission and access of the electronic medical record.

[0120] A COVID-19 lesion prediction system based on smart contract and self-attention, including a medical record encryption module, an attention mechanism-based lesion segmentation module, a lesion prediction module, a combined scoring module, and a permission control module:

[0121] Medical record encryption module: patients introduce the functions of ACC, JC and DC into the cloud environment, configure ACC according to specific needs and access strategies, ensure that only authenticated and authorized visitors can access encrypted CT images, and provide secure and reliable electronic medical record access services for visitors;

[0122] Lesion segmentation module based on attention mechanism: the lesion segmentation module includes a neural network based on U-net, which replaces the original convolution operation with SwinT block, and inputs the CT image of the patient into the neural network module for down-sampling feature extraction and up-sampling feature learning. A multi-scale fusion unit (MSF) is designed at the terminal of the down-sampling to reshape the high-resolution features into token sequences consistent with the size and channels of the small-resolution features, and then model long-term dependencies through self-attention mechanism to learn global context features. At the same time, a mask difference unit (MDM) is used to obtain a mask score result of the segmentation network;

[0123] The lesion segmentation module includes:

[0124] SwinT block operation unit: the input image enters the neural network improved based on U-net network, and the convolution self-attention layer of each layer uses Swin Trasnformer for calculation;

[0125] Multi-scale fusion unit: during the input image pooling process, the high-resolution features that are constantly operated by self-attention are reshaped into token sequences consistent with the size and channels of the small-resolution features. After the token sequences are connected together, subsequent image segmentation and the next stage of lesion prediction module are performed respectively;

[0126] Mask difference unit: MDM first filters out the segmentation mask of the lesion image to be registered, and then subtracts the obtained mask pixel by pixel. Finally, MDM passes the result of subtraction through a sigmoid function to produce a probability between 0 and 1.

[0127] The COVID-19 lesion prediction system based on smart contract and self-attention, the lesion segmentation module based on attention mechanism takes U-net network as the feature extraction basis, introduces residual connection and attention mechanism, and uses SwinT block to replace the original convolution operation, so the feature extraction process of the network is as shown in the formula:

[0128] S=F Encoder +F Decoder

[0129] P i =C swin (P i-1 ),i=1,2,3,4

[0130] Ti = Concat(C cbr (T i+1 ),P j ),i = 1,2,3,4,j ≥ i A

[0131] F Encoder and F Decoder represent the down-sampling and up-sampling process of the segmentation network respectively, which learns the long-term dependency between the detailed features and the global context information of the image by global pooling and self-attention operation. Conv and Concat represent convolution layer and concatenation operation respectively; C swin includes pooling operation and Swin block, each pooling operation reduces the image pixels by half and increases the channel number by one; C cbr includes normalization operation and Relu activation function; P i and T i represent different stages to which the encoder and decoder belong respectively.

[0132] The lesion prediction module based on attention mechanism: make full use of the information aggregated after the pooling operation, and learn the dependency between channels by using attention mechanism;

[0133] The lesion prediction module includes:

[0134] Spatial self-attention unit of static context: use self-attention mechanism to integrate information between multiple slices.

[0135] Dynamic adaptive self-attention unit: better capture the context spatial relationship between lesions CT in the same period.

[0136] The spatial self-attention unit of static context in the COVID-19 lesion prediction system based on smart contract and self-attention: assuming that the input feature map is S, the feature maps S 1 , S 2 and S 3 are calculated by convolution, and then their dimensions are reshaped to R n×c , where n = h x w. Next, we multiply S 1 with the element-wise transpose of S 2 to get the weight map P. Similarly, we perform the same operation on the remaining feature maps to get the corresponding self-attention maps. The formal description is as follows:

[0137] P = F scale (W 1 Conv(S), W 2 Conv(S))

[0138] In the above formula, F scale(·) is an element-wise multiplication operation, and the size of the weighted feature map is the same as that of the original feature map. W represents different weights for calculating the convolution feature map, respectively. Then, the weighted sum operation is performed on multiple self-attention mappings, and a sigmoid activation function is applied to obtain the final spatial self-attention mapping P n1×n1 , where F sum represents the weighted sum:

[0139] S = W a3 Conv(S a ) + W q1 Conv(S q ) + W z3 Conv(S z )

[0140] F = F scale (Sigmod(F sum (P a , P q , P z )), S)

[0141] The dynamic adaptive self-attention unit of the COVID-19 lesion prediction system based on the smart contract and the self-attention first reshapes the feature size of the pooled fusion features of a single picture to d x 1 (d = c x h x w), and then calculates the attention weight of d x T to store the dynamic correlation between all period lesion features, so that the features are connected to the feature M d , and the formal description is as shown in the formula:

[0142] Q ij = SoftMax(F scale (W a M, Transpose(W b M)))

[0143] M SA = F scale (Q ij , W c M)

[0144] In the above formula, Softmax(·) represents a normalized exponential function, F scale (·) is an element-wise multiplication operation, and the size of the weighted feature map is the same as that of the original feature map. W A , W B , W C represent the feature layers M A , M B , M Cweighting; the feature map is input into an extended convolution layer, and batch normalization and ReLU are used to extend the receptive field. The obtained self-attention feature map is subjected to convolution operation by an adaptive temporal convolution kernel ATCK, and the feature information of the lesion is dynamically aggregated. The formal description is as shown in the formula denotes the ReLU activation function:

[0145]

[0146] M OUT = F scale (M SA ,M ATCK )

[0147] The combination score module: linearly combines the mask score after COVID-19 segmentation MDM processing and the prediction score of the prediction network to obtain the final COVID-19 infection probability score.

[0148] The permission control module: saves the prediction result and controls the access permission of the prediction result through the control contract.

[0149] Figure 9 is the comparison curve diagram of the experimental results of the COVID-19 course staging prediction system based on the attention mechanism of the present application and the other two methods. Among them, method 1 uses the traditional U-Net model for segmentation prediction without adding residual connection and SwinT block based on attention mechanism, method 2 uses a single segmentation model for pneumonia prediction without adding SCSSM and DASM modules; it can be seen that after adding the attention mechanism and multi-scale module learning, the prediction accuracy of COVID-19 is the highest, because the attention mechanism and multi-scale module learning constantly learn more useful features in the feature learning process, thereby improving the accuracy of the model in distinguishing COVID-19 and normal people.

[0150] It should be understood that, for those skilled in the art, improvements or changes can be made according to the above description, and all these improvements and changes shall belong to the protection scope of the appended claims of the present application.

Claims

1. A COVID-19 lesion prediction system based on smart contracts and self-attention, characterized in that, It includes a medical record encryption module, an attention-based lesion segmentation module, an attention-based lesion prediction module, a combined scoring module, and an access control module. The attention-based lesion segmentation module includes: SwinT block operation unit: The input image enters a neural network based on the U-net network improvement, and the convolutional self-attention layer of each layer is computed using the Swin Trasnformer; Multi-scale fusion unit: During the pooling process of the input image, the high-resolution features that continuously perform self-attention operations are reshaped into token sequences with the same size and channels as the low-resolution features. After the token sequences are connected together, they are used for subsequent image segmentation and the next stage of lesion prediction module. Mask Differential Unit: MDM first filters out the segmentation mask of the lesion image to be registered, and then subtracts the obtained mask pixel by pixel; finally, MDM passes the subtraction result through a sigmoid function to generate a probability between 0 and 1, which is used as the result score of the segmentation network. The lesion segmentation module includes a neural network based on U-net, replacing the original convolutional operations with SwinT blocks. The patient's CT images are input into this neural network module for downsampling feature extraction and upsampling feature learning. A multi-scale fusion unit (MSF) is designed at the downsampling terminal. This module learns global context features by modeling long-term dependencies through a self-attention mechanism, reshaping high-resolution features into token sequences with the same size and channels as low-resolution features, and then learning global context features by modeling long-term dependencies through a self-attention mechanism. Simultaneously, a masked difference unit (MDM) is used to process the segmentation results pixel-by-pixel to obtain a mask score result for the segmentation network. The lesion prediction module based on the attention mechanism: the pooling results of the segmentation network after the encoder are fed into the prediction network to perform the lesion prediction task; make full use of the information aggregated after the pooling operation, and use the attention mechanism to learn the dependencies between channels. The lesion prediction module includes: Spatial Self-Attention Unit (SCSSM) in Static Context: It takes multi-scale fusion features from a segmentation network as input and uses a self-attention mechanism to integrate information between multiple slices; it uses a self-attention mechanism to dynamically weight the multi-scale fusion features of each learned 2D slice; by suppressing irrelevant slices to encode spatial information, SCSSM can focus on and predict the most relevant features for COVID-19. Dynamic adaptive self-attention unit: better captures the contextual spatial relationship between lesions in CT scans at the same time. At the same time, the output of the self-attention unit will be passed through a sigmoid function to generate a probability between 0 and 1, which will be used as the prediction score of the prediction network. Combined scoring module: linearly combines the Mask scoring results after the COVID-19 segmentation MDM processing and the prediction scores of the prediction network; Access control module: Saves prediction results and controls access permissions to the prediction results through control contracts.

2. The COVID-19 lesion prediction system based on smart contracts and self-attention as described in claim 1, characterized in that, The attention-based lesion segmentation module uses the U-net network as the basis for feature extraction, introduces residual connections and attention mechanisms, and replaces the original convolution operation with a SwinT block. The feature extraction process of the network is shown in the formula: S=F Encoder +F Decoder P i =C swin (P i-1 ),i=1,2,3,4 T i =Concat(C cbr (T i+1 ),P j ),i=1,2,3,4,j≥iA In the above formula, F Encoder and F Decoder represents the downsampling and upsampling processes of the segmentation network, respectively. Through global pooling and self-attention operations on the input image, it learns the long-term dependencies between the image's detailed features and global contextual information; Conv and Concat represent convolutional layers and cascaded operations, respectively; C swin This includes pooling operations and Swing blocks; each pooling operation halves the image pixels and doubles the number of channels; C cbr Includes normalization operations and the ReLU activation function; P i and T i These represent the different stages to which the encoder and decoder belong.

3. The COVID-19 lesion prediction system based on smart contracts and self-attention as described in claim 1, characterized in that, The spatial self-attention unit of the static context: assuming the input feature map is S, the feature map S is calculated through convolution. 1 S 2 and S 3 Then reshape their dimensions to R n×c Where n = h × w; Next, we will S 1 With S 2 Multiplying the elements by their horizontal transposes yields the weight map P; similarly, we perform the same operation on the remaining feature maps to obtain the corresponding self-attention maps; their formal description is shown in the formula: P=F scale (W 1 Conv(S),W 2 Conv(S)) In the above formula, F scale (·) represents element-wise multiplication, and the weighted feature map has the same size as the original feature map; W represents the different weights for calculating the convolutional feature map; then, a weighted sum operation is performed on multiple self-attention maps, and the sigmoid activation function is applied to obtain the final spatial self-attention map P∈R. n1×n1 Its formal description is shown in the formula, where F sum This indicates a weighted summation: S=W a3 Conv(S a )+W q1 Conv(S q )+W z3 Conv(S z ) F=F scale (Sigmod(F sum (P a ,P q ,P z )),S)。 4. The COVID-19 lesion prediction system based on smart contracts and self-attention as described in claim 1, characterized in that, The aforementioned dynamic adaptive self-attention unit first reshapes the feature size of the pooling fusion features of a single image to d×1 (d=c×h×w), then calculates the attention weights of d×T to store the dynamic correlation between all periodic lesion features, thus connecting the features to feature M∈R. d In this context, its formal description is shown in the formula: Q ij =SoftMax(F scale (W a M,Transpose(W b M))) M SA =F scale (Q ij ,W c M) In the above formula, Softmax(·) represents the normalized exponential function, F scale (·) represents element-wise multiplication; the weighted feature map has the same size as the original feature map. A W B W C Representing feature layer M respectively A M B M C The weights; The feature map is input into an extended convolutional layer, and batch normalization and ReLU are used to expand the receptive field. The obtained self-attention feature map is convolved using an adaptive temporal convolutional kernel (ATCK) to dynamically aggregate the lesion's feature information. Its formal description is shown in the formula, where... Represents the ReLU activation function: M OUT =F scale (M SA ,M ATCK ); The output M of the prediction network out It uses a sigmoid function to generate a probability between 0 and 1, which is the Prediction Score obtained by the prediction network.

5. A COVID-19 lesion prediction system based on smart contracts and self-attention as described in claim 1, characterized in that, The medical record encryption module introduces the functions of smart contract Access Control Contract (ACC), smart contract JCContract (JC), and smart contract DC Contract (DC) into the cloud environment. ACC is configured according to specific needs and access policies to ensure that only authenticated and authorized users can access encrypted CT images, providing users with secure and reliable electronic medical record access services. Access Control Contract (ACC): This smart contract is written based on the information of the visitor and the relevant electronic medical records stored in the cloud; ACC is responsible for verifying the legitimacy of the access request and controlling access permissions to the electronic medical records according to the access policy. JC Contract: This smart contract is used to record and manage visitor behavior and reputation information; JC maintains the visitor's blocking time, handles visitor violations, and returns the visitor's blocking time. DC Contract: This smart contract is used for delegated control; DC maintains the relationship between the principal and the delegator, and records the principal's host address and status information.

6. A COVID-19 lesion prediction system based on smart contracts and self-attention as described in claim 5, characterized in that, The access control in the access control module includes: The patient invokes the DC smart contract to delegate the authority of electronic medical records to the delegator. At the same time, the patient uses their own private key to decrypt the electronic medical records to obtain the plaintext content of the electronic medical records. Only the authorized delegator and the patient themselves can view the system's judgment results. The patient uses a temporary key to encrypt the electronic medical record and digitally signs the encrypted electronic medical record using their own private key. After receiving the encrypted information, the principal uses their own private key to decrypt the electronic medical record and verifies the integrity of the electronic medical record using the patient's public key. Finally, the principal uses their own public key to re-encrypt the decrypted electronic medical record and upload it to cloud storage. When medical institution B needs to access a patient's electronic medical record, medical institution B invokes the corresponding smart contract for access control; the smart contract verifies the identity of medical institution B, the serial number of the electronic medical record, and the legality of the access behavior; if the verification is successful, medical institution B obtains access control permissions. Simultaneously, the client and medical institution B generate a temporary key; the client uses the temporary key to decrypt the encrypted electronic medical record and uses their private key to digitally sign the decrypted electronic medical record; medical institution B uses the client's public key to decrypt the electronic medical record and uses the patient's public key to verify the integrity of the electronic medical record. When electronic medical records are not entrusted, patients can directly share their electronic medical records with medical institution B without the involvement of the entrusting party.

Images