Smart medical intelligent diagnosis guidance method and system based on blockchain and multimodal data
By adopting blockchain and multimodal data technology in the guidance system, combining the significance area multi-scale evaluation, multi-convolution kernel face recognition and multiple evaluation speech recognition models, the problem of insufficient guidance accuracy and security is solved. , realizes a high accuracy and high reliability guidance process, ensuring the safety of patient information.
Patent Information
- Application Number
- CN202410961411.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-07-18
AI Technical Summary
The existing guidance system has insufficient guidance accuracy and safety, which leads to anxiety and waste of time, and there is a risk of data breaches, affecting hospital operations and patient privacy.
Using intelligent medical intelligent guidance method based on blockchain and multimodal data, patients' face images and voice signals are collected through audio and video equipment, and differential image denoising model and multi-convolution kernel are used for multi-scale evaluation of significance areas. The verification face recognition model and the multiple evaluation HMM and Transformer preferred speech recognition model combine blockchain technology for identity recognition and data storage to ensure the security and reliability of the diagnosis guide process.
It improves the accuracy of identification of patient identity information and speech recognition accuracy, ensures the safety and reliability of the diagnosis guide process, reduces the consumption of computing resources, and ensures the immutability and integrity of data through blockchain technology.
Smart Images

Figure CN118522427B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of intelligent medical guidance technology, and specifically relates to a smart medical intelligent medical guidance method and system based on blockchain and multimodal data. Background Art
[0002] With the development of science and technology, the application of medical guidance systems in hospitals is becoming more and more widespread. Medical guidance systems can help patients quickly and accurately locate the required clinics or service points through intelligent means, greatly improving the convenience and efficiency of medical treatment. However, the existing medical guidance systems still have many functional limitations, such as insufficient guidance accuracy, which not only affects patients' first impression of the hospital, but also may cause patients to have unnecessary anxiety and waste time because they cannot find the correct clinic. In addition, the security and reliability of the medical guidance system are also issues that need to be urgently addressed. If the medical guidance system fails or data is leaked at a critical moment, it will directly affect the normal operation of the hospital and the privacy rights of patients.
[0003] With the continuous advancement of artificial intelligence technology, the medical guidance system is developing in a more intelligent and precise direction. On the basis of making full use of multi-modal data, the medical guidance system can achieve richer and more accurate patient identification and services. In addition, the introduction of blockchain technology has brought new solutions to the security and reliability of the medical guidance system. The distributed nature of blockchain can ensure the immutability and integrity of patient information, thereby improving data security. Therefore, it is necessary to propose a smart medical intelligent guidance method and system based on blockchain and multi-modal data. Summary of the invention
[0004] In view of the above deficiencies in the prior art, the present invention provides a smart medical intelligent guidance method and system based on blockchain and multimodal data, which improves the recognition accuracy of patient identity information, ensures the accuracy of voice recognition, and ensures the security and reliability of the guidance process.
[0005] To achieve the above objectives, the present invention provides a smart medical intelligent diagnosis guidance method and system based on blockchain and multimodal data, comprising the following steps:
[0006] S1. Use the audio and video equipment on the medical guidance machine to collect the patient's facial image;
[0007] S2, using a difference image denoising model based on multi-scale evaluation of salient regions to denoise the face image to obtain a denoised face image;
[0008] S3, using a verification face recognition model based on multiple convolution kernels to identify the denoised face image and obtain the patient's identity information;
[0009] S4. After obtaining the patient's identity information, the audio and video equipment on the medical guidance machine sends a voice prompt signal to prompt the patient to describe the condition, and the audio and video equipment is used to automatically extract the patient's voice signal to obtain a voice signal describing the condition;
[0010] S5. Use the HMM and Transformer optimization speech recognition model based on multiple evaluations to recognize the voice signals describing the condition, automatically register the patient based on the speech recognition results, and visually output the registration results to the patient through the guidance machine.
[0011] As a preferred embodiment of the present invention, in S2, the difference image denoising model based on multi-scale evaluation of salient regions includes a common image denoising model and an image denoising model based on deep learning, wherein the common image denoising model includes a median filter model and a Gaussian filter model, and the image denoising model based on deep learning includes a DnCNN model and a FFDnet model;
[0012] For the collected face images, the fully convolutional network (FCN) is used to first perform saliency detection, then the salient areas of the face images are reconstructed at multiple scales, and the peak signal-to-noise ratio of the salient areas is detected at multiple scales.
[0013] If the peak signal-to-noise ratio of the significant area at each scale exceeds the set signal-to-noise ratio threshold, the ordinary image denoising model is directly used for denoising, otherwise the image denoising model based on deep learning is used for denoising;
[0014] Among them, the specific architecture of FCN for saliency detection includes:
[0015] Input layer: receives the original face image;
[0016] Convolutional layer sequence: multiple convolutional layers are set up to extract image features. These convolutional layers contain convolution operations, batch normalization, and ReLU activation functions;
[0017] Pooling layer: It is set after the convolution layer to reduce the spatial size of the feature map;
[0018] Transposed convolution layer: used to gradually restore the spatial size of the feature map and achieve upsampling;
[0019] Category Activation Map: The last layer is a convolutional layer with a sigmoid activation function to generate a probability map of each pixel belonging to a salient region;
[0020] Loss function: During training, use the cross entropy loss function or the Dice loss function to optimize the prediction of the saliency map;
[0021] Optimizer: Adam or SGD is used to update network weights.
[0022] As a preferred solution of the present invention, in S3, the verification face recognition model based on multiple convolution kernels adopts a face recognition model with a smoothing convolution kernel and a sharpening convolution kernel, and the face recognition model is FaceNet;
[0023] The recognition process is to recognize the denoised face image through the face recognition model without smoothing or sharpening, and obtain the recognition result A;
[0024] The denoised face image is filtered using a smoothing convolution kernel, and then the face recognition model is used to recognize the denoised face image after smoothing filtering to obtain a recognition result B;
[0025] The denoised face image is filtered using a sharpening convolution kernel, and then the denoised face image after sharpening and filtering is recognized using a face recognition model to obtain a recognition result C;
[0026] If the identification results A, B, and C are consistent, the patient's identity information is finally confirmed;
[0027] Among them, the specific architecture of FFDnet includes:
[0028] Input layer: receives the original denoised facial image;
[0029] Convolutional layer: Use convolution kernels of different sizes and types to extract features, including at least one smoothing convolution kernel and at least one sharpening convolution kernel;
[0030] Batch Normalization Layer: used to normalize the input, speed up the training process, and improve model stability;
[0031] Activation layer: ReLU layer is used to introduce nonlinearity to help the network learn more complex features;
[0032] Pooling layer: reduces the spatial dimension of features, extracts the main features, and reduces the amount of calculation;
[0033] Fully connected layer: In the deep layer of the network, the learned features are mapped to the final output;
[0034] Loss function: Triplet Loss is used to train the network to bring similar faces closer and push different faces farther away.
[0035] As a preferred embodiment of the present invention, in said S5, the HMM and Transformer optimization speech recognition model based on multiple evaluations includes a speech recognition model based on HMM and a speech recognition model based on Transformer;
[0036] For the speech signal describing the condition, the peak signal-to-noise ratio (PSNR) and the objective speech quality evaluation (PESQ) are used to evaluate the speech signal. If the evaluation result is higher than the set speech signal threshold, the speech recognition model based on HMM is used to recognize the speech signal describing the condition. Otherwise, the speech recognition model based on Transformer is used to recognize the speech signal describing the condition.
[0037] Among them, the specific architecture of the HMM-based speech recognition model includes:
[0038] Input layer: receives the preprocessed acoustic feature sequence, expressed as:
[0039] x t =Input(t); where t is the time step, x t is the acoustic eigenvector at time step t;
[0040] Feature extraction layer: Automatically learn acoustic feature representation and use CNN to automatically extract features, expressed as:
[0041] h t =CNN(x t ), where h t It is the feature representation after CNN processing;
[0042] Multi-scale time series analysis layer: Pooling layers with different window sizes are used to capture features of different time scales, expressed as:
[0043] ;
[0044] Among them, s represents different scales, To capture the feature representations at different time scales;
[0045] Deep HMM layer: Use LSTM to optimize state transition and emission probability, expressed as:
[0046] s t =LSTM(h t ,s t- 1); e t =Emission(s t );
[0047] Among them, s t is the hidden state of the HMM at time step t, e t is the corresponding emission probability, and the Emission function is used to convert the hidden state into the emission probability;
[0048] Attention mechanism layer: Use weighted sum to integrate the features of key frames, expressed as:
[0049] ;
[0050] ;
[0051] Among them, c t is the context vector at time step t, α ti corresponds to h i The attention weight of h i 、h j They represent the hidden state representations of the i-th and j-th time steps in the time series, respectively, and T is the sequence length;
[0052] Context modeling layer: uses an encoder-decoder structure to provide additional context information;
[0053] Output layer: Use the softmax layer to generate the final recognition result;
[0054] The specific architecture of the Transformer-based speech recognition model includes:
[0055] Input layer: receives acoustic feature sequences;
[0056] Feature encoding layer: A fully connected layer is used to map acoustic features into a high-dimensional space;
[0057] Position encoding layer: Use different frequencies of sine and cosine functions to encode position information and provide the model with position information in the time series;
[0058] Transformer layer: includes self-attention mechanism and position encoding, and uses self-attention mechanism to process encoded features. The self-attention mechanism is expressed as:
[0059] ;
[0060] Among them, Q, K, and V represent query, key, and value matrices respectively, and d k is the dimension of the key;
[0061] Context fusion layer: integrates the output of the Transformer layer;
[0062] Output layer: generates the final recognition result;
[0063] The loss function used is expressed as:
[0064] ;
[0065] Among them, y t is the one-hot encoding of the target label, is corresponding to y t probability.
[0066] As a preferred solution of the present invention, it also includes: S6, encoding the denoised face image using a dimensional geometric increase image coding model based on super-resolution reconstruction to obtain a denoised face image coding result.
[0067] As a preferred solution of the present invention, in S6, the dimension geometric increase type image coding model based on super-resolution reconstruction adopts a generative adversarial network GAN and an encoder;
[0068] Firstly, the denoised face image is super-reconstructed to obtain an ultra-high resolution denoised face image;
[0069] Then, the denoised face image is encoded by an encoder, and the encoding result is reconstructed into an image. If the distortion degree of the reconstructed image does not exceed the distortion threshold, the encoding result is retained as the final encoding result.
[0070] If the distortion degree of the reconstructed image exceeds the distortion threshold, the encoder is reused to double the encoding dimension, the denoised face image is encoded again, and the encoding result is reconstructed into an image. If the distortion degree of the reconstructed image does not exceed the distortion threshold, this encoding result is retained as the final encoding result. If it still exceeds the distortion threshold, this step is repeated until the distortion degree of the reconstructed image does not exceed the distortion threshold, and the final encoding result is retained.
[0071] Among them, the network structure of GAN for image super-resolution reconstruction specifically includes:
[0072] Encoder: A CNN that converts the low-resolution LR image into a low-dimensional feature representation;
[0073] Generator: The generator consists of several groups of upsampling layers and convolutional layers, which are used to upsample the encoder output and gradually reconstruct it into a high-resolution HR image;
[0074] Discriminator: A CNN that distinguishes whether the input is a real high-resolution image or a high-resolution image generated by the generator;
[0075] Loss function: including adversarial loss, which is used to train the adversarial process between the discriminator and the generator;
[0076] Reconstruction loss, using mean square error or perceptual loss, is used to ensure the quality of the reconstructed image;
[0077] Optimizer: Adam or SGD is used to update network weights.
[0078] As a preferred solution of the present invention, it also includes: S7, using blockchain technology to store the patient's identity information, the final encoding result of the denoised face image, the voice signal describing the condition, and the registered clinic information on the chain.
[0079] Among them, the signal-to-noise ratio threshold, the speech signal threshold and the distortion threshold can be specifically set according to requirements.
[0080] The smart medical intelligent diagnosis guidance system based on blockchain and multimodal data is used to implement the above-mentioned smart medical intelligent diagnosis guidance method based on blockchain and multimodal data, including audio and video equipment, denoising module, image recognition module and speech recognition module set on the diagnosis guidance machine:
[0081] The audio and video equipment is used to collect the facial image of the patient, and after obtaining the patient's identity information, send out a voice prompt signal to prompt the patient to describe the condition, and automatically extract the patient's voice signal to obtain a voice signal describing the condition;
[0082] A denoising module uses a difference image denoising model based on multi-scale evaluation of salient regions to denoise the collected facial images of patients and obtain denoised facial images;
[0083] The image recognition module uses a verification-based face recognition model based on multiple convolution kernels to recognize the denoised face image and obtain the patient's identity information;
[0084] The speech recognition module uses the HMM and Transformer optimization speech recognition model based on multiple evaluations to recognize the speech signals describing the condition, automatically register the patient based on the speech recognition results, and visually output the registration results to the patient through the guidance machine.
[0085] As a preferred solution of the present invention, it also includes a coding module, which uses a dimensional geometric increase image coding model based on super-resolution reconstruction to encode the denoised face image to obtain a denoised face image coding result.
[0086] As a preferred solution of the present invention, it also includes a blockchain module for storing patient identity information, final encoding results of denoised facial images, voice signals describing the condition, and registered clinic information on the chain.
[0087] The models and modules involved in the present invention can be executed by an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The functions of the above-mentioned models and modules are realized by executing the software by the processor.
[0088] The beneficial effects of the present invention are:
[0089] The present invention proposes a differential image denoising model based on multi-scale evaluation of salient regions to denoise face images. The model evaluates the salient regions of face images at multiple scales, and selectively uses a common image denoising model or an image denoising model based on deep learning according to the evaluation results, which not only ensures the denoising effect of face images, but also reduces the consumption of computing resources.
[0090] The present invention proposes a verification-type face recognition model based on multiple convolution kernels to recognize denoised face images. The model not only recognizes the original denoised face images, but also uses multiple convolution kernels to filter the denoised face images before recognition. When multiple recognition results are consistent, the patient's identity information is finally confirmed, which significantly improves the recognition accuracy of the patient's identity information.
[0091] The present invention proposes a HMM and Transformer optimization speech recognition model based on multiple evaluations to recognize speech signals describing the condition. The model uses multiple evaluation indicators to evaluate the speech signal. If the evaluation results are all good, it is considered that the quality of the speech signal is high, and the speech signal describing the condition is recognized using the HMM-based speech recognition model; otherwise, the speech signal describing the condition is recognized using the Transformer-based speech recognition model. This speech recognition solution not only ensures the accuracy of speech recognition, but also reduces the consumption of computing resources.
[0092] The present invention proposes a geometrically increasing image coding model based on super-resolution reconstruction to encode denoised face images. The model first reconstructs the image with super-resolution, and then encodes the image by geometrically increasing the dimension, ensuring that lower-dimensional coding can be achieved while ensuring coding accuracy, saving more storage space.
[0093] The present invention utilizes blockchain technology to store patient identity information, denoised facial image encoding results, condition description voice signals, and consultation room information on the chain, ensuring the security and reliability of the guidance process. BRIEF DESCRIPTION OF THE DRAWINGS
[0094] Figure 1 It is a flow diagram of the method of the present invention;
[0095] Figure 2 It is a functional block diagram of the system of the present invention. DETAILED DESCRIPTION
[0096] The embodiments of the present invention are further described below in conjunction with the accompanying drawings:
[0097] Example 1
[0098] like Figure 1As shown, the intelligent medical guidance method and system based on blockchain and multimodal data, when the patient comes to the guidance machine, the guidance method includes the following steps:
[0099] S1. Use the audio and video equipment on the medical guidance machine to collect the patient's facial image;
[0100] S2, using a difference image denoising model based on multi-scale evaluation of salient regions to denoise the face image to obtain a denoised face image;
[0101] S3, using a verification face recognition model based on multiple convolution kernels to identify the denoised face image and obtain the patient's identity information;
[0102] S4. After obtaining the patient's identity information, the audio and video equipment on the medical guidance machine sends a voice prompt signal to prompt the patient to describe the condition, and the audio and video equipment is used to automatically extract the patient's voice signal (this part can be achieved based on the known audio and video equipment) to obtain a voice signal describing the condition;
[0103] S5. Use the HMM and Transformer optimization speech recognition model based on multiple evaluations to recognize the speech signal of the condition description, automatically register the patient according to the speech recognition result, and visually output the registration result to the patient through the guidance machine; after registration, the patient can directly go outside the corresponding clinic to wait for treatment;
[0104] S6. Encoding the denoised face image using a dimensional geometrically increasing image coding model based on super-resolution reconstruction to obtain a denoised face image coding result;
[0105] S7. Using blockchain technology, the patient’s identity information, the final encoding result of the denoised face image, the voice signal describing the condition, and the registered clinic information are stored on the chain.
[0106] Since the voice signal occupies relatively little storage space, there is no need to encode the voice signal describing the condition.
[0107] Differential image denoising takes into account the differences between different regions of the image during the image denoising process and applies different denoising strategies to different regions, which does not consume excessive computing resources and can achieve better denoising effects.
[0108] In S2, the difference image denoising model based on multi-scale evaluation of salient regions includes a common image denoising model and an image denoising model based on deep learning, wherein the common image denoising model includes a median filter model and a Gaussian filter model, and the image denoising model based on deep learning includes a DnCNN model (Deep Convolutional Neural Network for Image Denoising) and a FFDnet model (Fast and Flexible Deblurring Network); the common image denoising model and the image denoising model based on deep learning can adopt publicly known models.
[0109] For the collected face images, the fully convolutional network FCN (convolutional neural network CNN or region proposal network RPN can also be used) is used to first perform saliency detection, then multi-scale reconstruction of the salient areas of the face images is performed, and the peak signal-to-noise ratio of the salient areas is detected at multiple scales;
[0110] If the peak signal-to-noise ratio of the significant area at each scale exceeds the set signal-to-noise ratio threshold, the ordinary image denoising model is directly used for denoising, otherwise the image denoising model based on deep learning is used for denoising.
[0111] For example, a network structure of FCN capable of saliency detection includes:
[0112] Input layer: receives the original face image;
[0113] Convolutional layer sequence: multiple convolutional layers are used to extract image features. These convolutional layers contain convolution operations, batch normalization, and ReLU activation functions;
[0114] Pooling layer: It is set after the convolution layer to reduce the spatial size of the feature map;
[0115] Transposed convolution layer: used to gradually restore the spatial size of the feature map and achieve upsampling;
[0116] Class Activation Map (CAM): The last layer is a convolutional layer with a sigmoid activation function to generate a probability map of each pixel belonging to a salient region;
[0117] Loss function: During training, use cross entropy loss function or Dice loss function to optimize the prediction of saliency map;
[0118] Optimizer: Adam or SGD is used to update network weights.
[0119] In S3, the verification face recognition model based on multiple convolution kernels uses a face recognition model with smoothing convolution kernels and sharpening convolution kernels. The face recognition model is FaceNet (a deep convolutional neural network model for face recognition and clustering. CNN or RPN can also be used).
[0120] The recognition process is to recognize the denoised face image through the face recognition model without smoothing or sharpening, and obtain the recognition result A;
[0121] The denoised face image is filtered using a smoothing convolution kernel, and then the face recognition model is used to recognize the denoised face image after smoothing filtering to obtain a recognition result B;
[0122] The denoised face image is filtered using a sharpening convolution kernel, and then the denoised face image after sharpening and filtering is recognized using a face recognition model to obtain a recognition result C;
[0123] If recognition result A, recognition result B, and recognition result C are consistent, the patient's identity information is finally confirmed.
[0124] For example, a FaceNet network structure with smooth convolution kernels and sharp convolution kernels includes:
[0125] Input layer: receives the original denoised facial image;
[0126] Convolutional layer: uses convolution kernels of different sizes and types (including smoothing and sharpening) to extract features;
[0127] Batch Normalization Layer: used to normalize the input, speed up the training process, and improve model stability;
[0128] Activation layer: ReLU layer is used to introduce nonlinearity to help the network learn more complex features;
[0129] Pooling layer: reduces the spatial dimension of features, extracts the main features, and reduces the amount of calculation;
[0130] Fully connected layer: In the deep layer of the network, the learned features are mapped to the final output;
[0131] Loss functions: such as Triplet Loss, which is used to train the network to bring similar faces closer and push different faces further away.
[0132] In S5, the HMM and Transformer optimization speech recognition model based on multiple evaluations includes a speech recognition model based on HMM (Hidden Markov Model) and a speech recognition model based on Transformer;
[0133] The quality assessment of speech signals can be achieved through a variety of objective evaluation indicators, which are designed to quantify the quality of speech signals and reflect the perception of the human ear as much as possible. Peak signal-to-noise ratio (PSNR) and objective speech quality evaluation (PESQ) are two commonly used evaluation indicators, and their evaluation can be achieved based on well-known technologies.
[0134] For the medical condition description speech signal, the peak signal-to-noise ratio PSNR and objective speech quality evaluation PESQ are first used to evaluate the speech signal. If the evaluation result is higher than the set speech signal threshold, the HMM-based speech recognition model is used to recognize the medical condition description speech signal. Otherwise, the Transformer-based speech recognition model is used to recognize the medical condition description speech signal.
[0135] Among them, the specific architecture of the HMM-based speech recognition model includes:
[0136] Input layer: receives the preprocessed acoustic feature sequence, expressed as:
[0137] x t =Input(t); where t is the time step, x t is the acoustic eigenvector at time step t;
[0138] Feature extraction layer: Automatically learn acoustic feature representation and use CNN to automatically extract features, expressed as:
[0139] h t =CNN(x t ), where h t It is the feature representation after CNN processing;
[0140] Multi-scale time series analysis layer: Pooling layers with different window sizes are used to capture features of different time scales, expressed as:
[0141] ;
[0142] Among them, s represents different scales, To capture the feature representations at different time scales;
[0143] Deep HMM layer: Use LSTM to optimize state transition and emission probability, expressed as:
[0144] s t =LSTM(h t ,s t- 1); e t =Emission(s t );
[0145] Among them, s tis the hidden state of the HMM at time step t, e t is the corresponding emission probability, and the Emission function is used to convert the hidden state into the emission probability;
[0146] Attention mechanism layer: Use weighted sum to integrate the features of key frames, expressed as:
[0147] ;
[0148] ;
[0149] Among them, c t is the context vector at time step t, α ti corresponds to h i The attention weight of h i 、h j They represent the hidden state representations of the i-th and j-th time steps in the time series, respectively, and T is the sequence length;
[0150] exp is the exponential function, Score is the score function;
[0151] Context modeling layer: uses an encoder-decoder structure to provide additional context information;
[0152] Output layer: Use the softmax layer to generate the final recognition result;
[0153] The specific architecture of the Transformer-based speech recognition model includes:
[0154] Input layer: receives acoustic feature sequences;
[0155] Feature encoding layer: A fully connected layer is used to map acoustic features into a high-dimensional space;
[0156] Position encoding layer: Use different frequencies of sine and cosine functions to encode position information and provide the model with position information in the time series;
[0157] Transformer layer: includes self-attention mechanism and position encoding, and uses self-attention mechanism to process encoded features. The self-attention mechanism is expressed as:
[0158] ;
[0159] Among them, Q, K, and V represent query, key, and value matrices respectively, and d k is the dimension of the key;
[0160] The softmax function is a function that operates on a vector and can convert a real number vector into a probability distribution;
[0161] Context fusion layer: integrates the output of the Transformer layer;
[0162] Output layer: generates the final recognition result;
[0163] The loss function used is expressed as:
[0164] ;
[0165] Among them, y t is the one-hot encoding of the target label, is corresponding to y t probability.
[0166] This design enables the model to not only accurately recognize speech content, but also has the ability to handle long-distance dependencies and parallel computing, comprehensively improving the effect of speech recognition.
[0167] In S6, the dimension-increasing image coding model based on super-resolution reconstruction uses a generative adversarial network GAN and an encoder;
[0168] First, the denoised face image is reconstructed with super resolution (which can be achieved based on GAN) to obtain an ultra-high resolution denoised face image;
[0169] Then, an encoder (such as an autoencoder AE) is used to encode the denoised face image, and the encoding result is reconstructed into an image. If the distortion degree of the reconstructed image does not exceed the distortion threshold, the encoding result is retained as the final encoding result;
[0170] If the distortion degree of the reconstructed image exceeds the distortion threshold, the encoder is reused, the encoding dimension (the dimension of the latent space output by the encoder) is doubled, the denoised face image is encoded again, and the encoding result is reconstructed into an image. If the distortion degree of the reconstructed image does not exceed the distortion threshold, this encoding result is retained as the final encoding result. If it still exceeds the distortion threshold, this step is repeated until the distortion degree of the reconstructed image does not exceed the distortion threshold, and the final encoding result is retained.
[0171] For example, a GAN network structure for image super-resolution reconstruction includes:
[0172] Encoder: A convolutional neural network (CNN) that converts the low-resolution (LR) image into a low-dimensional feature representation;
[0173] Generator: The generator consists of several groups of upsampling layers and convolutional layers, which are used to upsample the output of the encoder and gradually reconstruct it into a high-resolution (HR) image;
[0174] Discriminator: The discriminator is also a CNN, which is used to distinguish whether the input is a real high-resolution image or a high-resolution image generated by the generator;
[0175] Loss function: including adversarial loss, which is used to train the adversarial process between the discriminator and the generator;
[0176] Reconstruction Loss, using mean square error (MSE) or perceptual loss, is used to ensure the quality of the reconstructed image;
[0177] Other losses, such as feature loss or style loss, can also be included as needed to improve the visual effect of the generated image.
[0178] Optimizer: Adam or SGD is used to update network weights.
[0179] Example 2
[0180] like Figure 2 As shown, the smart medical intelligent guidance system based on blockchain and multimodal data is used to implement the smart medical intelligent guidance method based on blockchain and multimodal data in Example 1, including audio and video equipment, denoising module, image recognition module, voice recognition module, encoding module and blockchain module arranged on the guidance machine:
[0181] The audio and video equipment is used to collect the facial image of the patient, and after obtaining the patient's identity information, send out a voice prompt signal to prompt the patient to describe the condition, and automatically extract the patient's voice signal to obtain a voice signal describing the condition;
[0182] A denoising module uses a difference image denoising model based on multi-scale evaluation of salient regions to denoise the collected facial images of patients and obtain denoised facial images;
[0183] The image recognition module uses a verification-based face recognition model based on multiple convolution kernels to recognize the denoised face image and obtain the patient's identity information;
[0184] The speech recognition module uses the HMM and Transformer optimization speech recognition model based on multiple evaluations to recognize the speech signal describing the condition, automatically register the patient based on the speech recognition results, and visually output the registration results to the patient through the guidance machine;
[0185] The encoding module encodes the denoised face image using a dimensional geometric increase image encoding model based on super-resolution reconstruction to obtain a denoised face image encoding result;
[0186] The blockchain module is used to store patient identity information, final encoding results of denoised facial images, voice signals describing the condition, and registered clinic information on the chain.
[0187] For example, a specific method of setting up audio and video equipment includes:
[0188] High-definition camera capable of capturing facial images of patients and supporting high-resolution video streaming;
[0189] Integrated microphone array for collecting patient's voice signals;
[0190] A speaker system for issuing voice prompt signals;
[0191] The voice activity detection unit detects the voice segments and silent segments in the voice signal, filters out the invalid silent parts, extracts only the valid voice data, and obtains the voice signal describing the condition;
[0192] The medical guidance equipment also includes:
[0193] Data processing unit, an integrated computing module, such as a GPU or CPU, is used to implement the functions of the modules and units involved.
[0194] An interactive interface, such as a touch screen or other interactive interface, is used to visually output the registration results;
[0195] Communication unit, used for data transmission and chain storage, etc.;
[0196] Storage structure, used to store patient data and interaction records (local storage, not conflicting with on-chain storage).
Claims
1. A smart medical guidance method based on blockchain and multimodal data, characterized by The following steps are involved: S1. Use the audio and video equipment on the medical guidance machine to collect the patient's facial image; S2, using a difference image denoising model based on multi-scale evaluation of salient regions to denoise the face image to obtain a denoised face image; S3, using a verification face recognition model based on multiple convolution kernels to identify the denoised face image and obtain the patient's identity information; S4. After obtaining the patient's identity information, the audio and video equipment on the medical guidance machine sends a voice prompt signal to prompt the patient to describe the condition, and the audio and video equipment is used to automatically extract the patient's voice signal to obtain a voice signal describing the condition; S5. Use the HMM and Transformer optimization speech recognition model based on multiple evaluations to recognize the speech signal of the condition description, automatically register the patient according to the speech recognition result, and visually output the registration result to the patient through the guidance machine; In the above S5, the HMM and Transformer optimization speech recognition model based on multiple evaluations includes a speech recognition model based on HMM and a speech recognition model based on Transformer; For the speech signal describing the condition, the peak signal-to-noise ratio (PSNR) and the objective speech quality evaluation (PESQ) are used to evaluate the speech signal. If the evaluation result is higher than the set speech signal threshold, the speech recognition model based on HMM is used to recognize the speech signal describing the condition. Otherwise, the speech recognition model based on Transformer is used to recognize the speech signal describing the condition. Among them, the specific architecture of the HMM-based speech recognition model includes: Input layer: receives the preprocessed acoustic feature sequence, expressed as: x t =Input(t); where t is the time step, x t is the acoustic eigenvector at time step t; Feature extraction layer: Automatically learn acoustic feature representation and use CNN to automatically extract features, expressed as: h t =CNN(x t ), where h t It is the feature representation after CNN processing; Multi-scale time series analysis layer: Pooling layers with different window sizes are used to capture features of different time scales, expressed as: ; Among them, s represents different scales, To capture the feature representations at different time scales; Deep HMM layer: Use LSTM to optimize state transition and emission probability, expressed as: s t =LSTM(h t ,s t- 1);e t =Emission(s t ); Among them, s t is the hidden state of the HMM at time step t, e t is the corresponding emission probability, and the Emission function is used to convert the hidden state into the emission probability; Attention mechanism layer: Use weighted sum to integrate the features of key frames, expressed as: ; ; Among them, c t is the context vector at time step t, α ti corresponds to h i The attention weight of h i 、h j They represent the hidden state representations of the i-th and j-th time steps in the time series, respectively, and T is the sequence length; Context modeling layer: uses an encoder-decoder structure to provide additional context information; Output layer: Use the softmax layer to generate the final recognition result; The specific architecture of the Transformer-based speech recognition model includes: Input layer: receives acoustic feature sequences; Feature encoding layer: A fully connected layer is used to map acoustic features into a high-dimensional space; Position encoding layer: Use different frequencies of sine and cosine functions to encode position information and provide the model with position information in the time series; Transformer layer: includes self-attention mechanism and position encoding, and uses self-attention mechanism to process encoded features. The self-attention mechanism is expressed as: ; Among them, Q, K, and V represent query, key, and value matrices respectively, and d k is the dimension of the key; Context fusion layer: integrates the output of the Transformer layer; Output layer: generates the final recognition result; The loss function used is expressed as: ; Among them, y t is the one-hot encoding of the target label, is corresponding to y t probability; The method further includes: S6, encoding the denoised face image using a dimension geometrically increasing image coding model based on super-resolution reconstruction to obtain a denoised face image coding result; In S6, the dimension geometric increase image coding model based on super-resolution reconstruction adopts a generative adversarial network GAN and an encoder; Firstly, the denoised face image is super-reconstructed to obtain an ultra-high resolution denoised face image; Then, the denoised face image is encoded by an encoder, and the encoding result is reconstructed into an image. If the distortion degree of the reconstructed image does not exceed the distortion threshold, the encoding result is retained as the final encoding result; If the distortion degree of the reconstructed image exceeds the distortion threshold, the encoder is reused to double the encoding dimension, the denoised face image is encoded again, and the encoding result is reconstructed into an image. If the distortion degree of the reconstructed image does not exceed the distortion threshold, this encoding result is retained as the final encoding result. If it still exceeds the distortion threshold, this step is repeated until the distortion degree of the reconstructed image does not exceed the distortion threshold, and the final encoding result is retained. Among them, the network structure of GAN for image super-resolution reconstruction specifically includes: Encoder: A CNN that converts the low-resolution LR image into a low-dimensional feature representation; Generator: The generator consists of several groups of upsampling layers and convolutional layers, which are used to upsample the encoder output and gradually reconstruct it into a high-resolution HR image; Discriminator: A CNN that distinguishes whether the input is a real high-resolution image or a high-resolution image generated by the generator; Loss function: including adversarial loss, which is used to train the adversarial process between the discriminator and the generator; Reconstruction loss, using mean square error or perceptual loss, is used to ensure the quality of the reconstructed image; Optimizer: Adam or SGD is used to update network weights.
2. The smart medical intelligent diagnosis guidance method based on blockchain and multimodal data according to claim 1 is characterized by: In the above S2, the difference image denoising model based on multi-scale evaluation of salient regions includes a common image denoising model and an image denoising model based on deep learning, wherein the common image denoising model includes a median filter model and a Gaussian filter model, and the image denoising model based on deep learning includes a DnCNN model and a FFDnet model; For the collected face images, the fully convolutional network (FCN) is used to first perform saliency detection, then the salient areas of the face images are reconstructed at multiple scales, and the peak signal-to-noise ratio of the salient areas is detected at multiple scales. If the peak signal-to-noise ratio of the significant area at each scale exceeds the set signal-to-noise ratio threshold, the ordinary image denoising model is directly used for denoising, otherwise the image denoising model based on deep learning is used for denoising; Among them, the specific architecture of FCN for saliency detection includes: Input layer: receives the original face image; Convolutional layer sequence: multiple convolutional layers are set up to extract image features. These convolutional layers contain convolution operations, batch normalization, and ReLU activation functions; Pooling layer: It is set after the convolution layer to reduce the spatial size of the feature map; Transposed convolution layer: used to gradually restore the spatial size of the feature map and achieve upsampling; Category Activation Map: The last layer is a convolutional layer with a sigmoid activation function to generate a probability map of each pixel belonging to a salient region; Loss function: During training, use the cross entropy loss function or the Dice loss function to optimize the prediction of the saliency map; Optimizer: Adam or SGD is used to update network weights.
3. The smart medical intelligent diagnosis guidance method based on blockchain and multimodal data according to claim 1 is characterized by: In the S3, the verification face recognition model based on multiple convolution kernels adopts a face recognition model with a smoothing convolution kernel and a sharpening convolution kernel, and the face recognition model is FaceNet; The recognition process is to recognize the denoised face image through the face recognition model without smoothing or sharpening, and obtain the recognition result A; The denoised face image is filtered using a smoothing convolution kernel, and then the face recognition model is used to recognize the denoised face image after smoothing filtering to obtain a recognition result B; The denoised face image is filtered using a sharpening convolution kernel, and then the denoised face image after sharpening and filtering is recognized using a face recognition model to obtain a recognition result C; If the identification results A, B, and C are consistent, the patient's identity information is finally confirmed; Among them, the specific architecture of FFDnet includes: Input layer: receives the original denoised facial image; Convolutional layer: Use convolution kernels of different sizes and types to extract features, including at least one smoothing convolution kernel and at least one sharpening convolution kernel; Batch Normalization Layer: used to normalize the input, speed up the training process, and improve model stability; Activation layer: ReLU layer is used to introduce nonlinearity to help the network learn more complex features; Pooling layer: reduces the spatial dimension of features, extracts the main features, and reduces the amount of calculation; Fully connected layer: In the deep layer of the network, the learned features are mapped to the final output; Loss function: Triplet Loss is used to train the network to bring similar faces closer and push different faces farther away.
4. The smart medical intelligent diagnosis guidance method based on blockchain and multimodal data according to claim 1 is characterized in that: Also includes: S7. Using blockchain technology, the patient’s identity information, the final encoding result of the denoised face image, the voice signal describing the condition, and the registered clinic information are stored on the chain.
5. A smart medical intelligent diagnosis guidance system based on blockchain and multimodal data, used to implement the smart medical intelligent diagnosis guidance method based on blockchain and multimodal data according to any one of claims 1 to 4, characterized in that: It includes audio and video equipment, denoising module, image recognition module and speech recognition module installed on the medical guidance machine: The audio and video equipment is used to collect the facial image of the patient, and after obtaining the patient's identity information, send out a voice prompt signal to prompt the patient to describe the condition, and automatically extract the patient's voice signal to obtain a voice signal describing the condition; A denoising module uses a difference image denoising model based on multi-scale evaluation of salient regions to denoise the collected facial images of patients and obtain denoised facial images; The image recognition module uses a face recognition model based on multiple convolution kernels to recognize the denoised face image and obtain the patient's identity information; The speech recognition module uses the HMM and Transformer optimization speech recognition model based on multiple evaluations to recognize the speech signals describing the condition, automatically register the patient based on the speech recognition results, and visually output the registration results to the patient through the guidance machine.
6. The smart medical intelligent diagnosis guidance system based on blockchain and multimodal data according to claim 5 is characterized by: The invention also comprises a coding module, which encodes the denoised face image by using a dimension geometrically increasing image coding model based on super-resolution reconstruction to obtain a denoised face image coding result.
7. The smart medical intelligent diagnosis guidance system based on blockchain and multimodal data according to claim 6 is characterized by: It also includes a blockchain module, which is used to store patient identity information, final encoding results of denoised facial images, voice signals describing the condition, and registered clinic information on the chain.
