Symptom preliminary screening and pre-diagnosis platform based on visual interaction and dynamic model updating

Abstract

The invention provides a symptom preliminary screening and pre-diagnosis platform based on visual interaction and dynamic model updating, and relates to the technical field of auxiliary medical judgment, the symptom preliminary screening and pre-diagnosis platform comprises a visual interaction module, a voice recognition module, an HIS docking module, a dynamic model updating module and an emergency processing module, and a pre-diagnosis guide system based on a visual scheme is adopted. In the step-in interval, the analysis and processing of characterization characteristics can be directly realized, and after the post case data and the comprehensive analysis and comparison of the past medical history are combined, a preliminary diagnosis and treatment process can be realized when a patient is in contact with the platform, so that the diagnosis and treatment efficiency is greatly improved. Various matching schemes such as excessive dependence on oral illness states or interactive illness state selection and a traditional so-called detection platform that a user needs to frequently rotate, act and the like are avoided, and the convenience of the user is firstly improved.

Description

Technical Field

[0001] The present invention relates to the field of auxiliary medical judgment technology, and specifically to a symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating. Background Art

[0002] According to the 3D medical guidance method, device, electronic device and medium based on ThreeJs disclosed in the Chinese patent publication number "CN114530237A", the method includes: obtaining the identity information of the user, the identity information at least includes gender and age, generating and displaying the human body 3D model image corresponding to the gender and age through ThreeJs, based on the user's touch operation on the displayed human body 3D model image, identifying the human body part information corresponding to the touch operation on the human body 3D model image, and obtaining the symptom description information input by the user, inputting the human body part information and the symptom description information into the trained convolutional neural network model, outputting the information of the department that can be booked, and determining the target department for booking according to the information of the department that can be booked and the user's needs. The invention has carried out a part of the pre-diagnosis plan, but it is still based on the active oral description of the personnel to obtain the disease situation, and then make a judgment. The fundamental logic is still a subjective judgment of the traditional disease-doctor. However, as a medical behavior, the patient's oral description depends largely on the condition of the disease. However, for the representation data, the invention cannot extract or process and optimize it, and its pre-diagnosis process is too lengthy, resulting in that the medical efficiency will not be improved compared with the conventional manual pre-diagnosis.

[0003] However, according to the existing technologies such as CN116052858A, an intelligent medical guidance method based on BERT and feature fusion, CN117079778A, a medical guidance method, system, storage medium and equipment based on knowledge graph, and CN117877701A, a hospital intelligent medical guidance method, device, terminal and medium, passive information input is still used, and medical resources cannot be fully utilized. Summary of the invention

[0004] Technical issues solved

[0005] In view of the deficiencies of the prior art, the present invention provides a symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating, which solves the problem of passive recommendation in the prior art.

[0006] Technical Solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating, the symptom screening and pre-diagnosis platform comprising:

[0008] The visual interaction module is used to collect images of the user and obtain information about facial features, limb postures, and epidermal lesions through image processing algorithms;

[0009] The speech recognition module is used to transcribe and analyze the user's speech in real time and supports dialect adaptation;

[0010] HIS docking module, used to connect to the hospital information system and obtain the user's medical history, test results and doctor scheduling information;

[0011] Dynamic model update module, used to compare the system's initial screening results with the final diagnosis feedback, and continuously optimize the combined CNN algorithm;

[0012] The emergency response module is used to detect or identify emergency situations and link emergency and AED equipment to provide emergency rescue support to users.

[0013] Preferably, the visual interaction module includes a depth camera and an infrared camera, which are used to perform multi-angle imaging of the user under different lighting conditions, and combine with a skeleton point detection algorithm to perform preliminary identification of restricted limb movement or abnormal posture.

[0014] Preferably, the speech recognition module adopts an acoustic model and a language model based on deep learning, and performs online / offline fine-tuning by collecting local dialect data to improve the accuracy of localized recognition.

[0015] Preferably, the dynamic model updating module includes a feedback data collection unit and a model training unit, wherein the feedback data collection unit is used to store the doctor's diagnosis results and the patient's actual medical record information; the model training unit performs training based on the feedback data regularly or in real time, so that the accuracy and adaptability of the initial symptom screening are continuously improved.

[0016] Preferably, the emergency response module is interconnected with the AED equipment and emergency system in the hospital in real time. When cardiac arrest, highly suspected stroke or other critical illness signs are detected, an alarm is automatically generated to prompt nearby personnel to use the AED, and the emergency department is notified to intervene.

[0017] Preferably, the symptom screening and pre-diagnosis platform also includes: a multimodal information fusion unit, which is used to unify the modeling of the image features output by the visual recognition module, the text features output by the speech recognition module, and the medical record information output by the hospital information system, and then perform symptom screening and pre-diagnosis decisions after fusing the multimodal features through the attention mechanism or Transformer structure.

[0018] Preferably, the symptom screening and pre-diagnosis platform further includes an infrared temperature measurement and smart bracelet connection module during the deployment process; when the temperature measurement or wristband physiological monitoring data (heart rate, blood pressure, blood oxygen, etc.) exceeds the safety threshold, the system automatically triggers the emergency processing module and sends a prompt to the user to seek medical treatment nearby or call for help.

[0019] Preferably, the symptom screening and pre-diagnosis platform uses differential privacy or homomorphic encryption to process sensitive data; anonymizes or extracts features of the user's facial images and epidermal feature images locally or at the edge before uploading; and sets up hierarchical access control to allow only authorized medical staff to view images or medical records, further ensuring the security and compliance of patients' personal health information.

[0020] Preferably, the application process of the symptom screening and pre-diagnosis platform includes the following steps:

[0021] S1: User access and identity verification: Open the platform at the self-service terminal or mobile terminal in the outpatient hall to obtain the user's basic identity information; if the user has retained information in the hospital information system (HIS), complete secondary verification through facial recognition, face comparison or mobile phone verification code to ensure compliance with access rights;

[0022] S2: Visual information collection: The platform collects external information such as the user's posture, facial features, and skin features through the configured camera or depth sensor, and performs preliminary processing on the image at the front end or edge side before encrypting and transmitting it to the back end server;

[0023] S3: Voice input and dialect recognition: Users can describe their main symptoms by voice. The platform uses an acoustic model and language model based on deep learning for voice transcription, and combines it with a dialect database for adaptive recognition to extract key disease keywords from semantics.

[0024] S4: Multimodal data fusion analysis: The platform fuses visual recognition output (facial and epidermal features, etc.) with speech / NLP output (symptom description text) and medical history information (if matched), and uses the improved convolutional neural network algorithm or attention mechanism to make preliminary judgments on symptoms and recommend departments.

[0025] S5: Dynamic model matching and HIS linkage: The platform calls the hospital information system (HIS) interface to obtain the user's previous diagnosis records, examination reports and allergy history, and performs model matching in combination with the current multimodal recognition results; if an abnormal or high-risk signal is identified, the system can trigger an emergency mode;

[0026] S6: Output of initial screening and pre-diagnosis results: The platform generates initial screening recommendations based on the above analysis, including: recommended departments or possible diseases; corresponding doctor scheduling information and registration portal; estimated waiting time and waiting information; if emergency signs are detected, instruct to go to the emergency department immediately or initiate AED linkage;

[0027] S7: Feedback collection and dynamic update: After the user completes the doctor's consultation, the platform compares the doctor's final diagnosis information with the initial screening results. If there is a deviation between the two, the deviation and the reason will be written into the feedback database; the platform iteratively updates the weights and rules in the multimodal model through a regular or real-time small batch training mechanism to continuously improve the recognition accuracy and triage quality.

[0028] Beneficial Effects

[0029] The present invention provides a symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating. It has the following beneficial effects:

[0030] 1. The present invention adopts a pre-diagnosis and guidance system based on a visual solution. When the user uses it, the analysis and processing of the characterization characteristics can be directly realized within the step-in interval. Combined with the post-case data and comprehensive analysis and comparison of the previous medical history, the patient can have a preliminary diagnosis and treatment process when he or she contacts the platform, which greatly shortens the lengthy pre-diagnosis time caused by the traditional passive acceptance solution. According to the current conventional pre-diagnosis time of 10-15 minutes, the average time from the user starting to use the platform to the end of using the platform to determine the diagnosis and treatment department is less than 5 minutes, avoiding excessive reliance on oral description of the condition or interactive selection of the condition and the traditional so-called detection platform requiring the user to frequently rotate, move and other coordination solutions, first of all, improving the convenience of the user.

[0031] 2. The present invention adopts a feedback optimization scheme, which can allow the platform system to continuously learn the optimization scheme under the condition of continuous use by users. When the amount of data and usage is sufficient, this product can evolve continuously and gradually replace a large part of the manual guidance work. As a machine system, its individual error rate can further achieve better accuracy in pre-diagnosis and guidance under the condition of continuous learning, further reducing the pressure on pre-diagnosis and guidance personnel. With the additional configuration of some intelligent products, even in some environments, the initial guidance and pre-diagnosis work can be cancelled, and the patient can be directly assigned to the corresponding consulting room, which greatly shortens the consultation time, improves the consultation efficiency, and further improves the utilization rate of medical resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the present invention;

[0033] Figure 2 It is a platform composition diagram of the present invention;

[0034] Figure 3 It is a schematic diagram of the outpatient clinic location of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Specific embodiment one:

[0037] As shown in the figure, the symptom screening and pre-diagnosis platform based on visual interaction and dynamic model update is characterized by: the visual interaction module of the symptom screening and pre-diagnosis platform includes a depth camera and an infrared camera, which are used to perform multi-angle imaging of the user under different lighting conditions, and combine the skeleton point detection algorithm to perform preliminary identification of limited limb movement or abnormal posture. The speech recognition module adopts an acoustic model and a language model based on deep learning, and collects local dialect data for online / offline fine-tuning to improve the accuracy of localized recognition. The dynamic model update module includes a feedback data acquisition unit and a model training unit, wherein the feedback data acquisition unit is used to store the doctor's diagnosis results and the patient's actual medical record information; the model training unit is trained regularly or in real time based on the feedback data, so that the accuracy and adaptability of the symptom screening are continuously improved. The emergency treatment module is connected to the AED equipment and emergency system in the hospital in real time. When cardiac arrest, highly suspected stroke or other critical signs are detected, an alarm is automatically generated and prompts nearby personnel to use AED, and the emergency department is notified to intervene. The visual interaction module is used to collect images of the user and obtain facial features, limb postures, and epidermal lesion information through image processing algorithms;

[0038] The visual interaction module includes the following components:

[0039] Depth camera, using models: Intel Real Sense D435, Microsoft Azure Kinect, captures the user's depth information and generates 3D point cloud data for human posture estimation and epidermal lesion detection using a USB 3.0 interface;

[0040] Infrared camera, using models: FLIR infrared camera, SeekThermal, to capture the user's skin temperature changes in low-light or no-light environments, and assist in identifying abnormal signs (fever, burns);

[0041] High-resolution RGB camera, using models: Logitech Brio, Sony Alpha series, to capture high-resolution images of the user's facial features and epidermal lesions;

[0042] Computing unit configuration: CPU: multi-core high-performance processor (Intel i9, AMD Ryzen 9), GPU: NVIDIA RTX 3080 and above graphics card, memory: at least 32GB RAM, storage: 2T and above SSD and 10T and above HDD are used to store image data and models, process data from each camera in real time, and execute image processing and feature extraction algorithms.

[0043] The network connection is configured with Gigabit Ethernet or high-speed Wi-Fi module to ensure low-latency, high-bandwidth data transmission between the camera and the computing unit.

[0044] The operation process is as follows:

[0045] Sp1. Data collection: The user stands at the self-service terminal or the entrance of the clinic. The depth camera captures the user's 3D posture and body shape information. The infrared camera monitors the user's skin temperature and identifies abnormal temperature areas. The RGB camera captures high-resolution facial and skin images.

[0046] Sp2. Data preprocessing: Image denoising and enhancement: Apply Gaussian filtering and other methods to remove image noise and improve image quality. Image alignment: Align images captured by multiple cameras to a unified coordinate system to ensure data consistency. Data compression and encryption: Perform JPEG compression and AES encryption on image data to ensure transmission efficiency and data security.

[0047] Sp3, feature extraction and analysis: Use depth camera data to estimate human posture and detect key points (such as shoulders, elbows, and knees). Use RGB cameras to extract facial features (such as facial key points and expression changes) and epidermal lesion features (such as rashes and ulcers). Use infrared camera data to analyze epidermal temperature distribution and identify abnormal temperature areas.

[0048] The skeleton point detection algorithm uses the human pose estimation algorithm OpenPose, HRNet, and AIphaPose based on the convolutional neural network (CNN). The human pose estimation is based on the generation of heatmaps and the positioning of key points. Suppose the input image is After extracting features through CNN, a heat map of each key point is generated For key point k, its heat map H k It can be expressed as:

[0049] Among them, .μ x ,μ y / is the actual coordinate of the key point in the image, σ is the standard deviation of the Gaussian distribution, and the heat map represents the position of each key point in the image through the Gaussian distribution. CNN learns to generate these heat maps through training so that the key point coordinates can be extracted from the heat map during prediction.

[0050] The facial feature recognition algorithm uses the face detection and recognition algorithm FaceNet and ArcFace. Face recognition is based on the distance measurement in the embedding space. Suppose the input face image is I f ,The loss function of FaceNet generated by CNN embedding vector Defined as:

[0051] in:

[0052] is the anchor point sample, is a positive sample of the same type as the anchor point, is a negative sample of a different class from the anchor point, α is the boundary, usually set to 0.2, [·] + Indicates a non-negative value.

[0053] TripletLoss learns effective face embedding representation by minimizing the distance between the anchor point and the positive sample and maximizing the distance between the anchor point and the negative sample.

[0054] The epidermal lesion detection algorithm uses image segmentation and classification algorithms U-Net, MaskR-CNN, and ResNet. U-Net is used for epidermal lesion detection. The input image Output segmentation mask M∈{0,1} H×W The loss function of U-Net is a combination of cross entropy loss and Dice loss:

[0055] in:

[0056] y i is the true label, p i is the prediction probability, N is the amount of collected data, the cross entropy loss measures the classification error of each pixel, and the Dice loss measures the overlap between the predicted segmentation and the true segmentation. The combination of the two improves the accuracy and robustness of the segmentation.

[0057] The temperature anomaly detection algorithm uses heat map analysis and anomaly detection algorithms, threshold-based detection, and machine learning anomaly detection models. Assuming the temperature distribution of the infrared image is T(x, y), the detection result A(x, y) of the abnormal area is defined as:

[0058] T tresold is the preset temperature threshold.

[0059] By comparing the temperature value of each pixel with the threshold, the high temperature area is identified as the abnormal area.

[0060] The visual interaction module captures the multi-dimensional image information of the user through multiple cameras and extracts key features through deep learning algorithms. These features include human posture, facial expressions, epidermal lesions and temperature abnormalities. The multimodal fusion unit conducts further analysis to assist in the initial screening and pre-diagnosis of symptoms.

[0061] The speech recognition module is used to transcribe and analyze the user's speech in real time and supports dialect adaptation;

[0062] The speech recognition module consists of the following parts:

[0063] High-quality microphones such as ShureSM7B and BlueYeti are used to collect clear voice input, reduce background noise interference, and retain XLR and USB interfaces, depending on the specific device.

[0064] The computing unit shares the computing unit with the visual interaction module, or configures a dedicated speech processing server to execute speech preprocessing, speech recognition and semantic parsing algorithms.

[0065] The operation process is as follows:

[0066] Sp1. Voice collection: The user uses the microphone of the self-service terminal or mobile device to input voice and describe the symptoms (such as "I have a headache", "I have severe back pain these days"). The microphone collects the voice signal and transmits it to the voice recognition module for processing.

[0067] Sp2, speech preprocessing: Apply spectral subtraction and other filtering algorithms to reduce background noise, eliminate echoes through adaptive filtering algorithms, improve speech signal quality, and enhance the clarity and intelligibility of speech signals.

[0068] Sp3, speech transcription and semantic analysis: Use deep learning acoustic models to transcribe clear speech signals into text, parse the text through natural language processing (NLP) technology, and extract key symptom keywords and disease descriptions.

[0069] The acoustic model uses a deep learning-based speech recognition model, DeepSpeech, wav2vec2.0, and Conformer. The deep learning-based acoustic model usually uses an end-to-end architecture to transform the speech signal X = (x1, x2, ..., x T ) is mapped to the text sequence Y = (y1, y2, ..., y U ), the loss function of CTC is:

[0070] p(Y|X) is the probability of generating the output text sequence Y given the input speech signal X. CTC allows the model to be trained without alignment annotations and maximizes the probability of the correct text sequence by calculating the sum of the probabilities of all possible alignment paths.

[0071] The language model adopts a language model based on deep learning, uses the Transformer architecture for language modeling, and defines a self-attention mechanism:

[0072] Q: query matrix, K: key matrix, V: value matrix, d k : The dimension of the key, used to scale the dot product, the self-attention mechanism calculates the dot product of the query and all keys to obtain the attention weight, and then multiplies it with the value matrix to generate a weighted output. This allows the model to focus on important information at different positions in the input sequence.

[0073] The dialect adaptation algorithm uses a multilingual model based on transfer learning, and adopts a transfer learning strategy to adapt it to the data of a specific dialect by fine-tuning the pre-trained acoustic model.

[0074] Fine-tuning process:

[0075] θ: parameters of the pre-trained model, θ′: model parameters after fine-tuning, η: learning rate, Loss function, f(X; θ): The predicted output of the model. The model parameters are adjusted through the gradient descent method to make it perform better on the corpus of a specific dialect and improve the accuracy of speech recognition.

[0076] The semantic parsing algorithm uses conditional random fields (CRF) for named entity recognition, defines the transfer matrix A and the emission matrix B, and the CRF model probability:

[0077] Y: label sequence, X: input sequence, f k : characteristic function, λ k : weight parameter, Z(X): normalization factor, T is the parameter amount of the conditional database, K is the parameter amount of the input feature; CRF calculates the conditional probability of the labeled sequence by combining multiple feature functions and weight parameters, which is suitable for sequence labeling tasks, such as extracting symptom keywords.

[0078] The speech recognition module collects the user's voice input through a high-quality microphone, and after noise elimination, echo cancellation and speech enhancement, uses an acoustic model to transcribe the voice signal into text. Subsequently, the language model and semantic parsing algorithm extract key symptom keywords and support dialect adaptation to ensure that voice input in different regional dialects can also be accurately recognized and parsed.

[0079] HIS docking module, used to connect to the hospital information system and obtain the user's medical history, test results and doctor scheduling information;

[0080] HIS docking modules include:

[0081] Servers and network equipment, high-performance servers are used to run HIS docking interfaces and data processing services, and dedicated networks are connected to the hospital intranet to ensure the high speed and security of data transmission.

[0082] Middleware devices include API gateways and message queue servers (such as Kafka and RabbitMQ), which manage and schedule data exchange with the HIS system to ensure orderly and reliable transmission of data streams.

[0083] The operation process is as follows:

[0084] Sp1. The platform establishes a data exchange interface with the hospital HIS system through standardized protocols (such as FHIR and HL7) and defines API endpoints for requesting and receiving data such as the patient's medical history, test results, and doctor scheduling information.

[0085] Sp2. Identity matching and verification: Perform face comparison through the visual recognition module to obtain the user's unique identifier (such as patient ID), and perform multiple identity authentication based on the mobile phone verification code or ID number entered by the user to ensure the accuracy and security of data matching.

[0086] Sp3. Data acquisition and update: After identifying the user's identity, the HIS interface is automatically called to obtain information such as their medical history, test results, etc. During the initial screening and pre-diagnosis process, the patient's diagnosis and treatment data is dynamically acquired or updated as needed to ensure the real-time and accuracy of decision-making.

[0087] The data exchange protocol uses FHIR to define the structure of medical data through resources. Each resource R i It can be expressed as:

[0088] Among them, identifier is a unique identifier, type specifies the resource type (such as Patient, Observation), and attributes are specific properties (such as name, age, medical history, etc.). FHIR achieves data interoperability between different systems through structured resource definitions, ensuring accurate transmission and analysis of medical data.

[0089] The identity matching algorithm uses a probability-based matching algorithm and multi-factor authentication (MFA), and uses the Bayesian formula to calculate the probability of identity matching:

[0090] P(Identity|Data): posterior probability of identity matching given data, P(Data|Identity): probability of data occurrence under the premise of correct identity, P(Identity): prior probability of identity, P(Data): total probability of data occurrence. By calculating the posterior probability and combining multiple verification factors (such as face comparison and verification code), the accuracy and security of identity matching can be improved.

[0091] The data synchronization and consistency algorithm uses a two-phase commit protocol (2PC) to define a node set N and a log entry set L. The leader replicates the log entry l to all slave nodes N. i , and wait for the majority of nodes to confirm before submitting:

[0092] I is the indicator function, if the node N i Confirm log entry l, then I(N i Confirmation l) = 1, otherwise 0. The confirmation of the majority of nodes ensures the consistency of log entries and the high availability of the system to prevent data loss or conflict.

[0093] The HIS docking module establishes an interface with the hospital information system through standardized data exchange protocols to automatically obtain key information such as the user's medical history, test results, etc. Combined with visual recognition and multiple identity authentication, it ensures accurate matching and secure transmission of data, providing comprehensive data support for symptom screening and pre-diagnosis.

[0094] Dynamic model update module, used to compare the system's initial screening results with the final diagnosis feedback, and continuously optimize the combined CNN algorithm;

[0095] The dynamic model update module consists of the following parts:

[0096] Computing units, high-performance GPU servers, are used for model training and inference, and large-capacity storage devices (such as NAS, SAN) are used to store training data and model parameters.

[0097] Data storage devices and database servers (such as MySQL and PostgreSQL) are used to store feedback data and training samples, and regularly back up data to offline storage or cloud storage to ensure data security and recoverability.

[0098] The operation process is as follows:

[0099] Sp1. Feedback data collection. After the user completes the doctor's face-to-face consultation, the system records the doctor's final diagnosis, the confirmed department, the prescribed examination items and other information, compares these data with the platform's initial screening results, and records deviations and possible reasons;

[0100] Sp2. Model training and optimization: The collected feedback data is used as a new training sample and input into the model training unit. The model is trained and fine-tuned using the improved combinatorial CNN algorithm, the model parameters are optimized, and incremental learning or transfer learning strategies are adopted to ensure that the new data can be effectively integrated into the existing model, thereby improving the adaptability and accuracy of the model.

[0101] Sp3. Model deployment and iteration: After the trained and optimized model is verified, it is deployed to the production environment to replace or assist the existing diagnostic model. A version management strategy is adopted to record the version number and changes of each model update to ensure the traceability and stability of model iteration. A / B testing is performed regularly to evaluate the performance and effect of the new model to ensure that the update does not introduce negative effects.

[0102] The input feature map is defined by combining the convolutional neural network (CNN) and the attention mechanism structure. Extract high-level features through convolutional layers:

[0103] Among them, Θ is the convolution kernel parameter, F is the input feature map, H is the height, W is the width, and C is the number of channels (the number of channels of RGB images is 3). Conv(F; Θ) is a convolution operation, which convolves the input feature map F with the convolution kernel Θ to generate a new feature map F′.

[0104] Introduce the attention mechanism and calculate the weighted features through self-attention:

[0105] Q=W q F′, K=W k F ′ , V=W v F ′ , W q ,W k ,W v is the weight matrix, Q: query matrix, K: key matrix, V: value matrix, is the scaled dot product attention score, QK T To calculate the dot product of the query and the key, we get the similarity score between each query and all the keys, d k is the dimension of the key (i.e., the number of columns of K), which is used to scale the dot product result to prevent the dot product result from being too large in high-dimensional space, causing the gradient to vanish or explode.

[0106] Calculate separately score ij Represents the similarity score between the i-th query and the j-th key. The scaled point integral is passed through the soft max The function is normalized to get the attention weights AttentionWeights). The similarity scores are converted into probability distributions to indicate the degree of attention each query pays to different keys. Multiply the attention weights by the value matrix V to get the weighted output, Attention(Q,K,V) is the attention output of the i-th query.

[0107] The multi-head attention mechanism is used to enhance the model’s feature expression capability:

[0108] h: the number of attention heads. By parallelizing multiple attention heads, we can capture different subspace information of input features and enhance the expressiveness of the model. i : The output of the i-th attention head;

[0109] in, is the weight matrix of the i-th head, used to linearly transform query, keys, and values.

[0110] Concat(head1,head2,…,head h ): Concatenate the outputs of all attention heads on the feature dimension, assuming the output dimension of each head is d v , then the dimension after splicing is h×d v , W O : Output weight matrix, used to linearly transform the spliced ​​multi-head output, where d model is the characteristic dimension of the model.

[0111] Linear transformation: Perform h different linear transformations on the input query, key, and value to obtain h different sets of Q i ,K i ,V i , parallel attention calculation: for each group Q i ,K i ,V i Apply the self-attention mechanism to get the head i , splicing and linear transformation: put all heads i Concatenate on the feature dimension and pass W O Perform a linear transformation to obtain the final multi-head attention output.

[0112] The combinatorial CNN algorithm extracts local features through convolutional layers and captures global dependencies through a multi-head attention mechanism, enhancing the expression of features and the decision-making ability of the model.

[0113] Incremental learning and transfer learning are algorithms for incremental learning and transfer learning based on existing models. The fine-tuning process in transfer learning is defined as:

[0114] θ: parameters of the pre-trained model, θ′: model parameters after fine-tuning, η: learning rate, Loss function, f(X; θ): The predicted output of the model. Incremental learning adapts to new data by keeping the parameters of some network layers unchanged and only updating the parameters of specific layers. Formula explanation: By adjusting the learning rate and selectively updating parameters, incremental learning allows the model to adapt to new data while retaining previous knowledge, thereby improving the generalization ability of the model.

[0115] The feedback learning mechanism adopts a feedback mechanism that combines supervised learning and reinforcement learning, and defines the reward function R based on the triage accuracy:

[0116] Optimize model parameters through policy gradient method:

[0117] α: learning rate, p(Y|X; θ): model output probability, the triage results are evaluated by the reward function, and the model parameters are adjusted using the policy gradient method to enhance the probability of correct triage and reduce the occurrence of incorrect triage.

[0118] The emergency response module is used to detect or identify emergency situations and link emergency and AED equipment to provide emergency rescue support to users.

[0119] The emergency response module includes the following components:

[0120] The emergency call button uses an emergency button device (Honeywell, Panasonic). When the user presses the button in an emergency, the emergency response process is immediately triggered and connected to the hospital's internal network or emergency response system.

[0121] AED (automated external defibrillator) equipment uses Philips Heart Start and Zoll AED Plus to provide the defibrillation treatment required for patients with cardiac arrest. It is installed and arranged in easily accessible locations such as the outpatient lobby, clinic entrance, and near the emergency department.

[0122] Emergency surveillance cameras use high-definition surveillance cameras (such as Axis and Hikvision) to monitor the user's status in real time, detect emergency signs (such as fainting and difficulty breathing), connect to the central monitoring system, and transmit video data in real time.

[0123] The emergency broadcast system uses the hospital's internal broadcast system (such as Telex, Bosch) to issue instructions and guidance information to on-site personnel when an emergency is detected.

[0124] The operation process is as follows:

[0125] Sp1. Emergency Detection: The emergency monitoring camera captures the user's behavior and posture in real time. The system detects abnormal behavior (such as sudden collapse, unconsciousness) through visual recognition algorithms. At the same time, the infrared camera monitors changes in skin temperature and identifies acute symptoms (such as pale skin and low temperature caused by cardiac arrest). When cardiac arrest, highly suspected stroke or other critical signs are detected, the system immediately triggers the emergency treatment process;

[0126] Sp2, AED linkage and emergency notification: The system locates the nearest AED device through the internal network, and guides on-site personnel to quickly obtain the AED through the display screen or indicator light. It automatically sends an alarm to the hospital emergency department to inform the specific location and situation of the emergency. When the emergency call button is triggered, it simultaneously notifies the emergency department and nearby medical staff to go to the scene;

[0127] Sp3. Emergency call and guidance: The emergency broadcast system issues instructions and step-by-step guidance on the use of AED to on-site personnel through audio and video. The emergency response module plays a demonstration video on the use of AED on the self-service terminal or display screen to help on-site personnel operate correctly.

[0128] Sp4. Monitoring and feedback: Key data during the first aid process (such as AED operation records and changes in patient physiological parameters) are recorded in real time and fed back to the platform. The emergency processing module incorporates relevant data into the dynamic model update module for subsequent algorithm optimization and emergency response strategy adjustment.

[0129] The abnormal behavior detection algorithm is based on the temporal behavior recognition model of CNN and LSTM, and YOLO is used for real-time target detection. YOLO (YouOnlyLookOnce) is used for real-time target detection, and the loss function is defined as:

[0130] λ coord ,λ noobj ,λ class : weight coefficient, S×S: grid division, B: number of bounding boxes predicted for each grid, Describes whether there is a target, x, y: bounding box center coordinates, C: bounding box confidence, p c : Category probability is not, YOLO performs target detection directly on the image through a single neural network. The loss function comprehensively considers the position, confidence and category prediction of the bounding box to optimize the detection accuracy and speed of the model.

[0131] The emergency priority judgment algorithm adopts a rule-based decision tree and a scoring-based priority algorithm, and defines a scoring function S:

[0132] s1,s2,…,s n: Scores of different factors (such as heart rate, blood oxygen, abnormal behavior), α, β, γ, …, δ: weights of each factor, through the weighted scoring function to comprehensively evaluate multiple factors, calculate the priority of the emergency event, and decide whether to trigger the emergency handling process.

[0133] The emergency response optimization algorithm uses Q-Learning to optimize the emergency response strategy and defines the Q function:

[0134] s: current state, a: current action, r: immediate reward, γ: discount factor, s ′ : next state, a′: next action, by updating the Q value, learning to take the best action in different states to optimize the emergency response strategy and improve the response efficiency and success rate.

[0135] The emergency response module monitors the user's behavior and physiological parameters in real time and uses abnormal behavior detection algorithms to identify potential emergency signs. Once an emergency is detected, the system immediately guides on-site personnel to obtain the AED device through the AED positioning and guidance algorithm, and uses the emergency event priority judgment algorithm to assess the severity of the event, and optimizes the emergency response strategy through the emergency response optimization algorithm to ensure a fast and effective rescue response.

[0136] The symptom screening and pre-diagnosis platform also includes: a multimodal information fusion unit, which is used to unify the modeling of the image features output by the visual recognition module, the text features output by the speech recognition module, and the medical record information output by the hospital information system. After fusing the multimodal features through the attention mechanism or Transformer structure, symptom screening and pre-diagnosis decisions are made.

[0137] The multimodal information fusion unit includes:

[0138] Computing unit: uses high-performance GPU servers to support multi-modal data processing and fusion algorithms;

[0139] Memory: At least 64 GB RAM to support real-time processing of large-scale data;

[0140] Storage: High-capacity SSD for storing multimodal data and fusion models;

[0141] High-speed internal network connection ensures low-latency data transmission between the visual interaction module, speech recognition module, HIS docking module and multimodal information fusion unit.

[0142] The operation process is as follows:

[0143] Sp1. Multimodal data reception: receive image feature data (facial features, limb posture, epidermal lesions) from the visual interaction module, receive text feature data (symptom description, keywords) from the speech recognition module, and receive medical record information (past medical history, test results) from the HIS docking module.

[0144] Unified feature modeling preprocesses the feature data output by each module (standardization and normalization), and uses feature embedding technology to convert the features of different modes into a unified high-dimensional vector representation:

[0145] F visual 、F audio 、F HIS are the features of the vision, speech and HIS modules respectively, and f embed is the feature embedding function;

[0146] Symptom screening and pre-diagnosis decision-making, input the fused comprehensive symptom description vector into the pre-trained classification model (such as multi-layer perceptron, deep neural network):

[0147] Among them, f class is a classification model, Θ is a model parameter, and the model outputs initial screening recommendations, including recommended departments, possible diseases, corresponding doctor information, etc.: Recommendation = argmax -C p(C|O) where C is a set of departments or diseases, and p(C|O) is the probability of each category.

[0148] The comprehensive classification algorithm uses a multi-layer perceptron (MLP) and a deep belief network (DBN). The output of the multi-layer perceptron is defined as:

[0149] W1, W2: weight matrix, b1, b2: bias vector, ReLU: activation function, softmax: output probability distribution function. Through multi-layer nonlinear transformation, MLP can learn complex mapping relationships, map the comprehensive symptom description vector to the probability distribution of each department or disease, and realize symptom screening and pre-diagnosis decision-making.

[0150] The multimodal information fusion unit uniformly models and fuses the multi-source feature data from the visual interaction module, speech recognition module, and HIS docking module, and generates a comprehensive symptom description vector after comprehensive analysis using the attention mechanism or Transformer structure. This vector is used for symptom screening and pre-diagnosis decision-making through a pre-trained classification model, and the corresponding department and doctor are recommended.

[0151] During the deployment process, the symptom screening and pre-diagnosis platform further includes infrared temperature measurement and smart bracelet connection modules; when the temperature measurement or wristband physiological monitoring data (heart rate, blood pressure, blood oxygen, etc.) exceeds the safety threshold, the system automatically triggers the emergency processing module and sends a prompt to the user to seek medical treatment nearby or call for help.

[0152] The infrared temperature measurement equipment uses FLIRT-Series infrared thermal imagers and Honeywell infrared temperature guns to monitor the user's body temperature in real time and identify abnormal signs such as fever;

[0153] The smart bracelet uses Fitbit, Apple Watch, and Garmin Vivosmart to monitor the user's physiological parameters, such as heart rate, blood pressure, blood oxygen, exercise volume, etc., connect to the platform via Bluetooth, and transmit data in real time.

[0154] The operation process is as follows:

[0155] Sp1. Data collection: When users wear smart bracelets at the self-service terminal or outpatient area, the device automatically connects and starts monitoring physiological parameters. The infrared temperature measurement device regularly scans the user's body temperature and records the data.

[0156] Sp2. Data monitoring and analysis. The platform receives real-time heart rate, blood pressure, blood oxygen and other data from smart bracelets, as well as body temperature data from infrared temperature measurement equipment. It detects abnormal physiological data through preset safety thresholds (such as heart rate that is too high or too low, blood oxygen below the normal range, fever, etc.).

[0157] Sp3. Abnormal data processing When any physiological parameter is detected to exceed the safety threshold, the system automatically triggers the emergency processing module and sends an emergency reminder to the user, suggesting that he or she seek medical treatment nearby or call for help immediately. At the same time, the abnormal data is sent to the medical team to ensure rapid response and processing.

[0158] Sp4. Data recording and feedback. All physiological data and abnormal events are recorded in the platform database for subsequent analysis and model training. The data of abnormal events are incorporated into the dynamic model update module to enhance the system's detection and response capabilities to similar events.

[0159] Physiological parameter anomaly detection algorithm Threshold-based detection algorithm and support vector machine (SVM) are used for anomaly classification. The threshold-based detection algorithm is defined as follows:

[0160] A i : Abnormal marker of physiological parameter i, P i : Current physiological parameter value, T i : The preset threshold of physiological parameter i, by comparing the physiological parameter value with the preset threshold, directly determine whether there is an abnormality. Support vector machine (SVM) is used for abnormal classification of multi-dimensional physiological parameters and optimizes the classification decision boundary:

[0161] subjecttoy i (w·φ(x i )+b)≥1-ξ i ,ξi ≥0

[0162] w: weight vector, b: bias term, C: penalty parameter, ξ i : Relaxation variable, φ(·): kernel function, SVM separates normal and abnormal data by finding the hyperplane with the largest margin and allows a certain error (through relaxation variables) to achieve efficient multi-dimensional anomaly detection.

[0163] The data fusion and comprehensive evaluation algorithm uses Bayesian network to fuse multi-variable data and define the joint probability distribution:

[0164] X1,X2,…,X n are physiological parameters, Parents(X i ) is node X i The Bayesian network models the dependency between physiological parameters through conditional probability distribution and comprehensively evaluates the abnormal conditions of multiple parameters.

[0165] The adaptive threshold adjustment algorithm adopts dynamic threshold adjustment and a prediction model based on historical data to define the adaptive threshold T i (t) changes with time:

[0166] μ i (t): average value of physiological parameter i at time t, σ i (t): standard deviation of physiological parameter i at time t, k: adjustment coefficient. By dynamically calculating the mean and standard deviation of physiological parameters, the threshold is adjusted to adapt to the physiological changes of the user and reduce false alarms and missed alarms.

[0167] The infrared temperature measurement and smart bracelet connection module monitors the user's physiological parameters and body temperature in real time, and uses the physiological parameter abnormality detection algorithm to promptly identify potential health risks. Once an abnormality is detected, the system immediately triggers an emergency response through the emergency processing module to ensure fast and effective rescue. At the same time, all abnormal data is recorded and fed back to the dynamic horizontal update module to continuously optimize the system's detection and response capabilities.

[0168] The symptom screening and pre-diagnosis platform uses differential privacy or homomorphic encryption to process sensitive data; the user's facial images and epidermal feature images are anonymized or feature extracted locally or at the edge before uploading; and hierarchical access control is set up to allow only authorized medical staff to view images or medical records, further ensuring the security and compliance of patients' personal health information.

[0169] The data privacy and security protection module includes the following components:

[0170] The security gateway uses Palo Alto Networks and Cisco ASA to monitor and protect the security of data transmission to prevent unauthorized access and data leakage.

[0171] Encryption devices use hardware encryption modules (such as HSM, hardware security module) to encrypt sensitive data, encrypt the storage and transmission of sensitive data, and ensure the security of data at rest and in transit.

[0172] The storage server is equipped with a high-security data storage server that supports encrypted storage and access control, and securely stores all sensitive data, including images, audio, medical records, etc.

[0173] The operation process is as follows:

[0174] Sp1. Data encryption and anonymization: The facial images and skin feature images collected by the user in the visual interaction module are anonymized or feature extracted locally or at the edge to remove or hide personal identification information. Differential privacy technology is used to perturb sensitive data to ensure that personal privacy is not leaked during data analysis and model training. The encrypted data is uploaded to the back-end server to ensure the security of the data during transmission and storage.

[0175] Sp2. Hierarchical access control: Through role-based access control (RBAC) policies, define the access rights of different users (such as medical staff, managers, and technicians), allow only authorized medical staff to access specific images or medical records, strictly restrict unauthorized personnel from accessing sensitive data, record and audit all data access behaviors, and ensure the traceability and compliance of data access.

[0176] Sp3. Data transmission security: HTTPS protocol is used throughout the entire process to encrypt data transmission to ensure the security of data in the network. VPN (virtual private network) or other secure communication protocols are used to protect data transmission between the hospital's internal network and the platform.

[0177] Sp4. Data backup and recovery: Regularly back up sensitive data stored on the server to ensure high availability and recoverability of data, and establish a disaster recovery plan to ensure that critical data and services can be quickly restored in the event of system failure or data loss.

[0178] The security audit and anomaly detection algorithm uses a Gaussian mixture model (GMM) for anomaly detection and defines the probability density function of each behavior pattern:

[0179] π k : Mixing coefficient of the kth Gaussian distribution, μ k : The mean of the kth Gaussian distribution, Σ k: The covariance matrix of the kth Gaussian distribution. By fitting the data distribution of normal behavior, GMM can calculate the probability of new behavior data. Behaviors with low probability are judged as abnormal, which assists in detecting potential security threats.

[0180] The data privacy and security protection module ensures the security of all sensitive data during transmission, storage and access through encryption, anonymization and hierarchical access control. Differential privacy and homomorphic encryption technologies are used to protect individual privacy, and data access rights are controlled through RBAC and ABAC policies. Combined with security auditing and anomaly detection algorithms, potential security threats are monitored and prevented to ensure system compliance and data security.

[0181] The overall hardware configuration of the system includes:

[0182] Self-service terminal equipment:

[0183] Touchscreen: High-resolution touchscreen (such as touch LCD or OLED display).

[0184] Camera system: integrated depth camera, infrared camera and RGB camera.

[0185] Microphone and speaker: used for voice interaction.

[0186] Processing unit: An embedded computing module or connected to a central server via a network.

[0187] Communication module: Wi-Fi, Ethernet or 4G / 5G module to ensure the stability of data transmission.

[0188] Mobile application equipment:

[0189] Platform: iOS and Android smartphones or tablets.

[0190] It provides mobile visual interaction, voice input, health data viewing and follow-up reminder functions, supports wireless connection methods such as Bluetooth and Wi-Fi, and connects to the hospital's internal network and platform server.

[0191] Doctor's equipment:

[0192] A high-performance PC or tablet computer running the doctor-side application, a high-definition monitor to display the patient's comprehensive information and diagnostic recommendations, a keyboard, mouse or stylus to facilitate doctor input and operation, and a seamless connection with the hospital's internal network and platform server to ensure real-time data access.

[0193] Management-side equipment: high-performance servers or management workstations, running management-side applications, multi-screen displays, displaying system statistical reports and operating indicators, interacting with the hospital's internal network and platform servers for data, and supporting management and decision-making functions.

[0194] The whole process is as follows:

[0195] Sp1. The user starts the platform through the self-service terminal or mobile App, performs identity authentication (facial recognition, face comparison), describes the symptoms through the touch screen or voice input, the system collects visual and voice data in real time, the multimodal information fusion unit conducts comprehensive analysis, generates initial screening and pre-diagnosis recommendations, the user confirms the registration information, and completes the registration and queuing process.

[0196] Sp2. Back-end processing: The back-end server receives multimodal data from the front-end, stores and processes the data, the dynamic model update module continuously optimizes the triage algorithm based on the feedback data, the HIS docking module obtains and updates the patient's medical history and medical information in real time, the emergency processing module monitors and responds to emergencies in real time, and links the hospital's internal system.

[0197] Sp3, data flow and communication: The front-end device transmits the collected multimodal data to the back-end server through a secure network connection. The back-end server interacts with the HIS system and emergency response system through API or message queue. The data storage device safely saves all collected data and ensures data security through encryption and access control. The dynamic model update module obtains feedback data from the storage, performs model training and optimization, and deploys the optimized model to the production environment. Specific embodiment 2:

[0199] Based on the solution of the specific embodiment 1, the application process of the symptom screening and pre-diagnosis platform based on visual interaction and dynamic model update includes the following steps:

[0200] S1: Visual information collection: The platform uses the configured camera and depth sensor to collect the user's body shape, facial features, skin features and other external representation information in real time when the user approaches the platform area, and performs preliminary processing on the image at the front end or edge side before encrypting and transmitting it to the back end server for processing;

[0201] S2: User access and identity authentication: Open the platform at the self-service terminal or mobile terminal in the outpatient hall to obtain the user's basic identity information; if the user has retained information in the hospital information system (HIS), complete secondary verification through facial recognition, face comparison or mobile phone verification code to ensure compliance with access rights;

[0202] S3: Voice input and dialect recognition: Users can describe their main symptoms by voice. The platform uses an acoustic model and language model based on deep learning for voice transcription, and combines it with a dialect database for adaptive recognition to extract key disease keywords from semantics.

[0203] S4: Multimodal data fusion analysis: The platform combines the visual information collection content output of S1 with the symptom description text and medical history information output of S3 through voice input and dialect recognition, and uses the improved convolutional neural network algorithm and Attention mechanism to make preliminary judgments on symptoms and recommend departments.

[0204] S5: Dynamic model matching and HIS linkage: The platform calls the hospital information system (HIS) interface to obtain the user's previous diagnosis records, examination reports and allergy history, and performs model matching in combination with the current multimodal recognition results; if an abnormal or high-risk signal is identified, the system can trigger an emergency mode;

[0205] S6: Output of initial screening and pre-diagnosis results: The platform generates initial screening recommendations based on the above analysis, including: recommended departments or pre-diagnosis diseases; corresponding doctor scheduling information and registration portal; estimated waiting time and waiting information; if emergency signs are detected, instruct to go to the emergency department immediately or initiate AED linkage;

[0206] S7: Feedback collection and dynamic update: After the user completes the doctor's consultation, the platform compares the doctor's final diagnosis information with the initial screening results. If there is a deviation between the two, the deviation and the reason will be written into the feedback database; the platform iteratively updates the weights and rules in the multimodal model through a regular or real-time small batch training mechanism to continuously improve the recognition accuracy and triage quality.

[0207] Case Description:

[0208] Patient A enters the hospital outpatient department and is recommended to enter the autonomous pre-screening platform usage area by volunteers or personal initiative (it should be clear that, in the previous case, the hospital and related social promotion work require volunteers or relevant staff to provide promotion explanations). The platform will then read the user's information through the camera, including his or her facial features, and further connect to the supporting related database to read the user's individual information, including his or her previous medical history and related disease information. The user's body shape, body surface, epidermis, face and other features are then read through the camera and entered into the database. When the user officially starts using the system, individual identification (including but not limited to fingerprints, faces, medical cards, medical cards, etc.) is required. After the identification is completed, the corresponding data resources are called, and then voice communication is conducted with the patient to confirm his or her medical needs. When the relevant needs are obtained, they are entered into the HIS system, and the corresponding pre-diagnosis results are output and hung in the corresponding consulting room. In addition, according to the severity of the situation, feedback is given to the corresponding consulting room to facilitate feedback on whether the corresponding priority situation needs to be provided. For details on the usage and equipment layout, please refer to the HIS system. Figure 3It should be noted that each platform is independent to avoid interference from multiple faces. The internal system can enter the faces of volunteers or service personnel separately as circumvention information to avoid misidentification when serving some older patients or inconvenient patients;

[0209] After the patient sees the doctor, the doctor will enter the relevant diagnosis into the hospital's data system. The backend of this platform will conduct a secondary comparison and learning of the patient information that enters the platform for diagnosis and treatment, determine the doctor's diagnostic criteria and the accuracy and basis of triage, and conduct further learning, including but not limited to the correctness of symptom inference, the correctness of triage, and the continuous learning of relevant diagnosis and treatment standards;

[0210] At the same time, in the case of localization, due to the existence of regional dialects, it is necessary to produce a dialect recognition database that conforms to the local language system through local information agencies in advance, so as to facilitate the use of local non-Mandarin speaking people. At the same time, due to the large changes in dialect areas, the system can enter the sound effect information when it cannot read the language, and hand it over to the background for secondary recognition, or the on-site service personnel can use it as a corrected version of the semantic input after error correction. In the case of continuous learning, subsequent identical voices will continue to expand the dialect library, so that dialect information can be recognized. In some special cases where recognition cannot be made, the on-site personnel will solve it manually or hand it over to the counter for processing.

[0211] The pre-diagnosis guidance system based on visual scheme is adopted. When the user uses it, the analysis and processing of the characterization characteristics can be directly realized within the step-in interval. Combined with the post-case data and the comprehensive analysis and comparison of the previous medical history, the patient can have a preliminary diagnosis and treatment process when he / she contacts the platform, which greatly shortens the lengthy pre-diagnosis time caused by the traditional passive acceptance scheme. According to the current conventional pre-diagnosis time of 10-15 minutes, the average time from the user starting to use the platform to the end of using the platform to determine the diagnosis and treatment department is less than 5 minutes. According to the National Health Commission of the People's Republic of China According to the Statistical Bulletin on the Development of my country's Health and Health Care Industry in 2023 released by the National Health and Medical Services Commission, the number of medical consultations in the country reached 9.55 billion that year, and the average daily workload of physicians reached 15.5 times. When the pre-diagnosis time of a single person is shortened, the physician's diagnostic efficiency will be greatly improved. If the pre-diagnosis is accurate, the physician's workload can be further reduced. At the same time, while reducing the pre-diagnosis labor, more space can be reserved for hospitals to build more medical spaces, increase the density of diagnostic physicians in hospitals, and greatly improve the hospital's radiation protection capabilities for medical treatment of surrounding residents, reduce the cost of new medical facilities, and improve the utilization rate of single buildings.

[0212] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprising a reference structure" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0213] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A symptom screening and pre-diagnosis platform based on visual interaction and dynamic model update, characterized by: The symptom screening and pre-diagnosis platform includes: The visual interaction module is used to collect images of the user and obtain information about facial features, limb postures, and epidermal lesions through image processing algorithms; The speech recognition module is used to transcribe and analyze the user's speech in real time and supports dialect adaptation; HIS docking module, used to connect to the hospital information system and obtain the user's medical history, test results and doctor scheduling information; Dynamic model update module, used to compare the system's initial screening results with the final diagnosis feedback, and continuously optimize the combined CNN algorithm; The emergency response module is used to detect or identify emergency situations and link emergency and AED equipment to provide emergency rescue support to users.

2. The symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating according to claim 1, characterized in that: The visual interaction module includes a depth camera and an infrared camera, which are used to perform multi-angle imaging of the user under different lighting conditions, and combine with a skeleton point detection algorithm to perform preliminary identification of restricted limb movement or abnormal posture.

3. The symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating according to claim 1, characterized in that: The speech recognition module adopts an acoustic model and a language model based on deep learning, and collects local dialect data for online / offline fine-tuning to improve the accuracy of localized recognition.

4. The symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating according to claim 1, characterized in that: The dynamic model updating module includes a feedback data collection unit and a model training unit, wherein the feedback data collection unit is used to store the doctor's diagnosis results and the patient's actual medical record information; The model training unit is trained regularly or in real time based on feedback data, so that the accuracy and adaptability of initial symptom screening are continuously improved.

5. The symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating according to claim 1 is characterized by: The emergency response module is connected to the AED equipment and emergency system in the hospital in real time. When cardiac arrest, highly suspected stroke or other critical illness signs are detected, an alarm is automatically generated to prompt nearby personnel to use the AED, and the emergency department is notified to intervene.

6. The symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating according to claim 1 is characterized by: The symptom screening and pre-diagnosis platform also includes: a multimodal information fusion unit, which is used to unify the image features output by the visual recognition module, the text features output by the speech recognition module, and the medical record information output by the hospital information system to model the image features, and then perform symptom screening and pre-diagnosis decisions after fusing the multimodal features through an attention mechanism or a Transformer structure.

7. The symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating according to claim 1 is characterized by: During the deployment process, the symptom screening and pre-diagnosis platform further includes an infrared temperature measurement and smart bracelet connection module; when the temperature measurement or wristband physiological monitoring data (heart rate, blood pressure, blood oxygen, etc.) exceeds the safety threshold, the system automatically triggers the emergency processing module and sends a prompt to the user to seek medical treatment nearby or call for help.

8. The symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating according to claim 1 is characterized by: The symptom screening and pre-diagnosis platform uses differential privacy or homomorphic encryption to process sensitive data; anonymizes or extracts features of the user's facial image and epidermal feature image locally or at the edge before uploading; and sets hierarchical access control to allow only authorized medical staff to view images or medical records, further ensuring the security and compliance of patients' personal health information.

9. The application process of the symptom screening and pre-diagnosis platform based on visual interaction and dynamic model updating according to any one of claims 1 to 8, characterized in that: The application process includes the following steps: S1: Visual information collection: The platform uses the configured camera and depth sensor to collect the user's body shape, facial features, skin features and other external representation information in real time when the user approaches the platform area, and performs preliminary processing on the image at the front end or edge side before encrypting and transmitting it to the back end server for processing; S2: User access and identity authentication: Open the platform at the self-service terminal or mobile terminal in the outpatient hall to obtain the user's basic identity information; if the user has retained information in the hospital information system (HIS), complete secondary verification through facial recognition, face comparison or mobile phone verification code to ensure compliance with access rights; S3: Voice input and dialect recognition: Users can describe their main symptoms by voice. The platform uses an acoustic model and language model based on deep learning for voice transcription, and combines it with a dialect database for adaptive recognition to extract key disease keywords from semantics. S4: Multimodal data fusion analysis: The platform combines the visual information collection content output of S1 with the symptom description text and medical history information output of S3 through voice input and dialect recognition, and uses the improved convolutional neural network algorithm and Attention mechanism to make preliminary judgments on symptoms and recommend departments. S5: Dynamic model matching and HIS linkage: The platform calls the hospital information system (HIS) interface to obtain the user's previous diagnosis records, examination reports and allergy history, and performs model matching in combination with the current multimodal recognition results; if an abnormal or high-risk signal is identified, the system can trigger an emergency mode; S6: Output of initial screening and pre-diagnosis results: The platform generates initial screening recommendations based on the above analysis, including: recommended departments or pre-diagnosis diseases; corresponding doctor scheduling information and registration portal; estimated waiting time and waiting information; if emergency signs are detected, instruct to go to the emergency department immediately or initiate AED linkage; S7: Feedback collection and dynamic update: After the user completes the doctor's consultation, the platform compares the doctor's final diagnosis information with the initial screening results. If there is a deviation between the two, the deviation and the reason will be written into the feedback database; the platform iteratively updates the weights and rules in the multimodal model through a regular or real-time small batch training mechanism to continuously improve the recognition accuracy and triage quality.

Images