Intelligent medical question and answer method and system based on multi-modal end-cloud collaboration

Abstract

The invention relates to the technical field of artificial intelligence and end-cloud collaboration, and particularly discloses an intelligent medical question answering method and system based on multi-mode end-cloud collaboration, and the method comprises the steps: collecting a visual image and voice data in real time through end-side equipment; generating a structured visual feature vector and a voice text feature vector; performing semantic alignment, and fusing into a multi-modal structured input packet with a uniform format; sending the input packet into a draft model to generate a plurality of parallel candidate drafts; transmitting the candidate draft, the draft model probability corresponding to the candidate draft and the input packet to cloud equipment; the target model verifies and finely adjusts the candidate drafts by adopting a multi-draft speculation sampling method based on importance weighting to generate a high-quality question and answer text; generating a digital human audio and video result; and pushing the digital human audio and video result to an end side device for display. According to the invention, the accuracy, real-time performance and user experience of intelligent medical question answering are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and edge-cloud collaboration, and in particular to an intelligent medical question-answering method and system based on multimodal edge-cloud collaboration. Background Technology

[0002] In recent years, artificial intelligence technologies, represented by Large Language Models (LLMs), have demonstrated enormous potential in the healthcare field, particularly in intelligent inquiry and health management. However, deploying computationally intensive LLMs directly on resource-constrained edge devices faces multiple challenges, including but not limited to performance bottlenecks, high inference latency, and user data privacy protection. While traditional pure cloud solutions possess powerful computing capabilities, their data transmission to the cloud may pose a risk of data privacy breaches, and their high requirements for network bandwidth and real-time performance make it difficult to meet the stringent requirements of low latency and high privacy in medical inquiry scenarios. On the other hand, pure edge deployment solutions are limited by the computing resources of edge devices, making it difficult to support the operation of large-scale LLMs, resulting in unsatisfactory question-answering quality and response speed.

[0003] To address these challenges, the industry has begun exploring edge-cloud collaborative AI architectures, aiming to combine the powerful computing capabilities of the cloud with the advantages of rapid response and data localization on the edge. However, existing edge-cloud collaborative solutions still have shortcomings in achieving efficient inference collaboration, ensuring data privacy, and providing a natural, high-fidelity interactive experience. For example, most solutions fail to fully utilize the preprocessing capabilities of edge devices to effectively reduce the burden on the cloud, and also fail to balance inference efficiency with the professionalism and accuracy of the generated content. Furthermore, how to present complex LLM outputs to users in a natural and reliable manner, especially through real-time interaction with anthropomorphic digital avatars, is a key issue that urgently needs to be addressed in the current field of intelligent healthcare.

[0004] Therefore, the existing technologies mainly have the following problems: 1) How to effectively solve the problem of deploying large language models on resource-constrained edge devices while meeting the requirements of low latency and high privacy in medical inquiry scenarios; 2) How to significantly reduce interaction latency while ensuring response quality through innovative inference acceleration mechanisms under the edge-cloud collaboration framework; 3) How to provide a natural and reliable visual interaction experience through a high-fidelity digital human engine, thereby improving the overall efficiency and user satisfaction of medical inquiry. Summary of the Invention

[0005] This invention aims to solve the aforementioned problems. To this end, this invention provides an intelligent medical question-answering method and system based on multimodal edge-cloud collaboration, effectively addressing the performance bottlenecks, high latency, and data privacy challenges of deploying large language models on resource-constrained edge devices, thereby improving the accuracy, real-time performance, and user experience of intelligent medical question answering.

[0006] This invention provides an intelligent medical question-answering method based on multimodal edge-cloud collaboration, the technical solution of which includes the following steps: S1: Real-time acquisition of visual images and voice data via edge devices; S2: Generate structured visual feature vectors from visual images; generate speech-text feature vectors from speech data; S3: Semantically align visual feature vectors and speech / text feature vectors, and fuse them into a unified format multimodal structured input package; S4: Feed the input packet into the draft model to generate multiple parallel candidate drafts; S5: Transmit the candidate drafts and their corresponding draft model probabilities, as well as the input packet, to the cloud device; the target model uses an importance-weighted multi-draft speculative sampling method to verify and fine-tune the candidate drafts, generating high-quality question-answer text; S6: Input high-quality question-and-answer text into the digital human engine to generate digital human audio and video results; S7: Push the digital human audio and video results to the edge device for display.

[0007] Furthermore, the importance-weighted multi-draft speculative sampling method is as follows: based on the probability distribution of the candidate drafts under the draft model and the target model, the weight corresponding to each candidate draft is calculated using the importance sampling algorithm, and the optimal intermediate draft is selected from the candidate drafts. Here, the draft model is a lightweight large language model, and the target model is a high-performance large language model; speculative verification is performed on the intermediate drafts.

[0008] Furthermore, for K candidate drafts ,in, This is the first candidate draft. The Kth candidate draft; the optimal intermediate draft Conditional distribution Defined as: in, This is a sequence of K candidate drafts sampled from the draft model. To select a draft given K candidate drafts. The probability of being the best intermediate draft; For each It satisfies the following constraints: and in, It is a set consisting of all possible candidate drafts; Selection probability for: in, Generate a draft for the target model The probability, Generate the i-th draft for the draft model. The probability, To obtain the minimum value, To obtain the maximum value; Maximizing is for all satisfying of It was carried out, and: if ,but .

[0009] Furthermore, the visual image includes a face image; a lightweight visual recognition model deployed on the edge device uses a convolutional neural network to extract gender and age label information from the visual image.

[0010] Furthermore, the voice data includes natural language queries; the voice recognition model deployed on the edge device adopts a non-autoregressive voice recognition framework to generate a vector representation containing semantics and emotion based on the voice data.

[0011] Furthermore, the input packet adopts JSON format and includes timestamps, semantic tags, user profile identifiers, and feature vectors, which are encapsulated through a lightweight encryption protocol.

[0012] Furthermore, the draft model is fine-tuned using knowledge distillation and 4-bit quantization compression, and based on a pre-trained dataset from the medical field. During the training phase, its loss function for: in, For conditional probability, To generate the i-th word in the sequence, To generate the first to (i-1)th words in the sequence, For multimodal input information, The sequence length is given.

[0013] Furthermore, in each decoding step, the draft model first performs Top-K truncation on the output distribution, and then performs vectorized parallel sampling and diversified bundle search on the truncated output distribution to obtain K candidate drafts.

[0014] Furthermore, the digital human engine is based on ER-NeRF technology.

[0015] This invention also provides an intelligent medical question-answering system based on multimodal edge-cloud collaboration, the technical solution of which includes: edge devices and cloud devices. The edge device is used to acquire visual images and speech data in real time; generate structured visual feature vectors based on visual images; generate speech-text feature vectors based on speech data; semantically align the visual feature vectors and speech-text feature vectors and fuse them into a unified format multimodal structured input package; send the input package into a draft model to generate multiple parallel candidate drafts; transmit the candidate drafts and their corresponding draft model probabilities, as well as the input package, to a cloud device; and display the digital human audio and video results. The cloud device is used to verify and refine the candidate drafts using a multi-draft speculative sampling method based on importance weighting, thereby generating high-quality question-and-answer text; inputting the high-quality question-and-answer text into the digital human engine to generate digital human audio and video results; and pushing the digital human audio and video results to the end device.

[0016] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: 1. This invention enables computationally intensive large language models to run efficiently on resource-constrained edge devices by deploying lightweight visual recognition models, speech recognition models, and lightweight large language models on the edge, and by employing optimization techniques such as model pruning, 4-bit quantization compression, and knowledge distillation. This overcomes the performance bottleneck caused by the excessive size of the model in traditional methods.

[0017] 2. This invention employs an innovative multi-draft speculative sampling mechanism based on importance weighting, making it particularly suitable for scenarios such as medical Q&A and health consultation. Medical text generation demands high accuracy and low illusion rate; traditional single-draft or weighted speculative sampling can easily lead to erroneous suggestions or ambiguous answers. This invention combines diversified draft generation on the client side with weighted verification in the cloud, enabling the system to quickly converge to semantically reasonable and medically logically consistent results. While maintaining low latency, it significantly improves the professionalism and credibility of the output. Combined with a client-side data localization processing mechanism, it effectively reduces the risk of privacy leaks and meets medical data security compliance requirements. Therefore, this method has significant medical application value and promotion potential.

[0018] 3. This invention, through multimodal information acquisition and preprocessing, can acquire users' visual images and voice data in real time, and integrate these multi-dimensional information into a structured input package, enabling subsequent language models to perform reasoning based on richer user profiles and contextual information, thereby generating more targeted, personalized and professional medical question-and-answer content.

[0019] 4. This invention performs preliminary processing and draft generation of user-sensitive multimodal information on the device side, effectively realizing localized processing of user data and significantly reducing the risk of privacy leaks that may occur when data is transmitted to the cloud. At the same time, the device-side preprocessing and draft generation also reduces reliance on cloud computing resources and network bandwidth, lowering operating costs.

[0020] 5. This invention integrates a digital human engine based on ER-NeRF technology. This engine can convert the final question-and-answer text into natural language speech signals in real time and drive the lip movements, facial expressions, and head posture of a high-fidelity digital human avatar to form a dynamic video stream synchronized with the speech. This anthropomorphic interaction method, combined with multilingual and personalized voice support, greatly enhances the user's immersion, trust, and satisfaction in medical inquiry scenarios.

[0021] 6. This invention constructs a complete closed-loop response process from user multimodal input, edge-cloud collaborative intelligent question answering, to anthropomorphic interactive display. The system supports multi-turn dialogue and context awareness, and provides personalized feedback through user profiling and intent recognition, offering an efficient, reliable, and continuously optimizable solution for intelligent medical assistants.

[0022] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the process provided by the present invention.

[0025] Figure 2 This is a schematic diagram of the overall structure provided by the present invention.

[0026] Figure 3 This is a schematic diagram of visual analysis provided by the present invention.

[0027] Figure 4This is a schematic diagram of speech analysis provided by the present invention.

[0028] Figure 5 This is a schematic diagram of speculative sampling provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0030] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0031] The following is combined Figures 1 to 5 The present invention will be further described in detail below, providing an intelligent medical question-answering method and system based on multimodal edge-cloud collaboration: In this embodiment, as Figure 1 and Figure 2 As shown, an intelligent medical question-answering method based on multimodal edge-cloud collaboration is provided, including the following steps: S1: With the user's consent, collect the user's visual images and voice data in real time through the terminal device.

[0032] The visual images include facial images, used to analyze the user's basic physiological information such as gender and age; the voice data includes the user's natural language queries, providing a foundation for subsequent speech recognition and semantic understanding. The acquisition process employs low-power optimization algorithms to ensure efficient operation of the edge devices.

[0033] S2: Generate structured visual feature vectors from visual images; generate speech-text feature vectors from speech data, which are vector representations containing semantics and emotion.

[0034] Lightweight visual and speech recognition models are deployed on edge devices. The visual recognition model, based on an efficient convolutional neural network or a lightweight Transformer architecture, performs visual analysis, detecting faces, classifying gender, and estimating age in facial images, generating structured visual feature vectors. The speech recognition model, based on an end-to-end neural network and incorporating an attention mechanism, performs speech recognition, transcribing speech data into text and extracting basic emotional features to generate vector representations containing semantics and emotion. The lightweight model is optimized through model pruning and 4-bit quantization compression techniques to adapt to resource-constrained edge device environments.

[0035] The lightweight visual recognition model employs the lightweight face detection network opencv_face_detector_uint8.pb and its structure definition file to generate a series of candidate bounding boxes and their corresponding confidence scores on the input image. The system sets a confidence threshold of 0.7 to filter out low-quality detection results, thereby ensuring the stability of the recognition. After detecting a facial region, the system automatically adds fixed-pixel contextual padding to each candidate bounding box and crops it to obtain a standardized face image input, thus completing the localization of the face position in the real-time video stream captured by the camera.

[0036] Subsequently, a convolutional neural network was used to extract gender and age labels from the face images. The face images were normalized to a 227×227 resolution, and a DNN blob (deep learning tensor container) was generated using the mean normalization parameters of the models. This blob was then fed in parallel into two models to predict the probability distributions of gender and age, respectively. Finally, the models used the index corresponding to the highest probability as the output category. The age component in visual analysis is as follows... Figure 3 As shown, during training, images labeled with age and images labeled with year are used, processed through convolutional feature mapping and fully connected layers. The loss function considers two types: a loss function based on age and a loss function based on age difference. The final loss function is the entropy loss function. Cross-entropy loss function and KL divergence distance Weighted sum: ,in, As the first weight, As the second weight, It is the third weight.

[0037] Lightweight speech recognition models employ a non-autoregressive speech recognition framework, such as... Figure 4As shown, the speech recognition model can transcribe natural language content in real time and can selectively output language tags, sentiment tags, and event tags simultaneously to enhance semantic context understanding. In the model preprocessing stage, the input waveform generates an 80-dimensional log-Mel filter bank (FBank), which is then downsampled by 6 times (LFR) and four embeddings—LID (language tag), SER (sentiment tag), AED (event tag), and ITN (inverse text regularization)—before being input into the SAN-M encoder. The SAN-M encoder captures short-term dependency features of the input sequence through a multi-layer self-attention mechanism; as the number of layers increases, the features diffuse, thereby improving the model's prediction accuracy. During training, the loss function considers the CTC loss function. and LID loss function SER loss function AED loss function wait.

[0038] S3: Semantically align visual feature vectors and speech / text feature vectors, and fuse them into a unified format multimodal structured input package. The input package uses JSON format and includes timestamps, semantic tags, user profile identifiers, and feature vectors, encapsulated using a lightweight encryption protocol to ensure data privacy and efficient transmission.

[0039] To ensure efficient collaboration between multimodal information recognition and inference, this embodiment further designs an interface communication mechanism. It interacts through a uniformly defined structured data format and standardized protocol, which not only improves the consistency of the system in the multimodal information processing process, but also provides good engineering support for subsequent model replacement and functional expansion.

[0040] The visual feature vectors are expressed in structured JSON, with fields including metadata such as gender (enumerated type), age, image resolution, and timestamp. The speech-text feature vectors are in standard text format (UTF-8 encoded natural language strings) and include a speech recognition confidence score for determining the reliability of the language model.

[0041] To align the semantic information of visual and speech modalities, cosine similarity is used to perform semantic fusion of modal features: in, The characteristics after fusion For visual feature vectors, This is a speech-text feature vector. Let be the magnitude of the vector. The fused features are normalized and mapped to a unified latent space to optimize the semantic consistency of subsequent model processing.

[0042] like Figure 2As shown, in this embodiment, multimodal information perception is realized at the edge sensing node. After the edge sensing node completes the data collection, analysis, identification and fusion, the obtained input packet is transmitted to the edge inference node through the local area network. The data format adopts a lightweight serialization protocol based on Protobuf to ensure transmission efficiency and decoding stability under low latency.

[0043] S4: The input packet is fed into the draft model deployed on the edge device to generate multiple parallel candidate drafts. The draft model is a lightweight large language model that supports the rapid generation of semantically coherent text answers through knowledge distillation and 4-bit quantization compression, and is fine-tuned based on a pre-trained dataset in the medical field.

[0044] A lightweight large language model has been specially optimized for this task. First, it is optimized through knowledge distillation and 4-bit quantization compression techniques, significantly reducing the parameter size and enabling efficient operation on resource-constrained edge devices. Second, the model is fine-tuned based on a massive pre-trained dataset in the medical field to ensure that the generated candidate content possesses semantic relevance and logical coherence within a medical context. During the training phase, the model minimizes the following loss function... To optimize its generation quality: in, Here, represents the conditional probability, indicating the model's confidence in predicting the current token. To generate the i-th word in the sequence, To generate the first to (i-1)th words in the sequence, For multimodal input information, The sequence length is given. Unlike traditional methods that generate only one optimal answer during the inference phase, the draft model of this invention is configured to output K high-probability candidate words or short sequences in parallel at each decoding step through techniques such as parallel sampling and diversified bundle search. The aim is to provide a diverse set of candidates for high-performance large language models in the cloud, thereby improving the efficiency and hit rate of subsequent speculative sampling mechanisms.

[0045] To ensure that the candidates generated by the draft model on the endpoint have both high confidence and diversity, this embodiment employs parallel sampling and diversified beam search in step S4 to output K candidate words or short sequences in parallel. Specifically, in each decoding step, the draft model first performs Top-K truncation on the output distribution, and then performs vectorized parallel sampling on the truncated output distribution to obtain K candidate drafts (or further expands each candidate into a short sequence). To improve the diversity among candidates, diversified beam search methods, such as group beam, diversity penalty, and stochastic beam methods, can be used.

[0046] The edge device will generate a list of each candidate token / short sequence (candidate draft) and its corresponding draft model probability. Send them to the cloud device together.

[0047] S5: Through an edge-cloud collaborative architecture, candidate drafts and their corresponding draft model probabilities, along with the input packet, are transmitted to the cloud device. The target model deployed on the cloud device uses an importance-weighted multi-draft speculative sampling method to verify and refine the candidate drafts, generating high-quality question-and-answer text and reducing redundant computation. The target model is a high-performance large language model.

[0048] This embodiment deploys a high-performance large language model in the cloud and employs an importance-weighted multi-draft speculative sampling method to verify and fine-tune the drafts. Traditional speculative sampling mechanisms sequentially verify all candidate drafts (candidate tokens) of the i-th sequence; however, this importance-weighted multi-draft speculative sampling method first calculates the probabilities of the draft models and target models of all candidate tokens in the i-th sequence, then calculates the importance weights and normalizes them to obtain the probability of each candidate token in the i-th sequence. Based on this probability, the optimal candidate token of the i-th sequence is selected; that is, only this one optimal candidate token needs to be verified. If rejected, the calculation is recalculated; if accepted, the candidate tokens of the (i+1)-th sequence are selected.

[0049] Figure 5 The basic process of a multi-draft speculative sampling method based on importance weighting is demonstrated.

[0050] First, the K candidate drafts generated in parallel by the end-side draft model are denoted as follows: ,in, This is the first candidate draft. This is the Kth candidate draft. All candidate drafts and their corresponding draft model probabilities are sent to the cloud.

[0051] Then, the high-performance large language model (target model) in the cloud performs a two-step verification process: Step 1: Intermediate token selection based on importance sampling. Unlike traditional speculative sampling strategies, this method's target model does not directly validate all K drafts, but instead first selects an intermediate token using a selection function. Choose the best intermediate draft from them. The selection process is based on weighted importance sampling. The weights take into account the factors considered in each draft. The probabilities under the draft model and the target model aim to maximize the likelihood of acceptance. Specifically, given a set of K candidate drafts... , Selected from Conditional distribution The definition of is: in, This is a sequence of K candidate drafts sampled from the draft model. Given K candidate drafts In this case, select draft. The probability of it being an intermediate draft.

[0052] For each It satisfies the following constraints: and in, This represents the set of all possible candidate drafts.

[0053] Selection probability for: in, Indicates the generation of a draft of the target model. The probability, This indicates that the draft model generates the i-th draft. The probability, To obtain the minimum value, To obtain the maximum value.

[0054] Furthermore, maximization is for all satisfied of It was carried out, and: if ,but .

[0055] Single draft speculation validation. The best intermediate draft will be selected. As the final draft, a standard speculative sampling validation process is performed. The probabilities of the target model and the draft model are compared, and a decision is made based on random numbers to determine whether to accept the model. Or refuse? If rejected, then from the modified probability distribution Resampling was performed on the selected intermediate drafts. Calculate its acceptance probability threshold : in, Generate target model The probability, Generate draft model The probability. Then, a number is randomly sampled from a uniform distribution. , ,if Then accept Otherwise, refuse. and from the corrected probability distribution Resampling. Defined as: in, for The corrected probability distribution, Generate the final draft for the target model The probability, Generate the final draft from the draft model The probability of.

[0056] like Figure 2 As shown, this embodiment achieves edge-cloud collaborative speculative inference by using serial prediction from the edge inference node and parallel verification from the cloud server, resulting in high-quality question-and-answer text. This high-quality question-and-answer text is then sent to a digital human engine, also deployed on the cloud server, where it enables visual interaction with the digital human.

[0057] S6: Input high-quality question-and-answer text into the digital human engine to generate digital human audio and video results.

[0058] The engine is based on ER-NeRF (Explicit-Rendered Neural Radiance Field) technology, which integrates explicit expression and neural rendering to generate high-fidelity video streams of medical-style digital humans; its speech synthesis module generates natural speech, supporting multiple languages ​​and personalized timbres; its dynamic rendering module drives lip movements, facial expressions and head posture through phoneme alignment to ensure high frame rates and low latency.

[0059] The ER-NeRF technology optimizes dynamic head representation through a three-plane hash representation and introduces a lightweight region attention mechanism. It uses multi-resolution region information to explicitly capture the relationship between dynamic features and spatial regions, and through external attention and cross-modal channel attention, it enables dynamic conditions to selectively act on specific facial regions.

[0060] S7: Pushes the audio and video results of the digital human to the end device for display through real-time streaming technology.

[0061] This embodiment can achieve multi-turn dialogue by repeating the above steps (S1-S7), and optimize the continuity of question and answer by combining the context awareness module; through user profiling and speech recognition, it can provide personalized feedback and improve the real-time performance, accuracy and user trust of medical inquiries.

[0062] In the execution flow of the digital human engine's visual interaction, semantic-driven parsing is performed first. Text and speech synthesis results are obtained from the language generation module, extracting semantic information (such as emotion, intonation, and keywords) and speech temporal features (such as phonemes, rhythm, and pauses) as control parameters for facial movements. Next is lip-syncing and expression mapping. A phoneme alignment algorithm maps text to phoneme-level temporal labels, combined with a pre-trained lip-sync action library (viseme-to-mesh), generating high-precision lip-sync and expression fusion control signals. Next is ER-NeRF expression-driven rendering. Explicit geometric templates acquired from multiple perspectives are loaded, and a neural volumetric rendering network is used to finely model facial expressions. The explicit mesh provides the basic shape and boundaries, while the NeRF network supplements dynamic texture changes (such as muscles and wrinkles) to achieve natural rendering. Finally, image generation and output convert the rendering results into 2D video frames in real time, fuses the speech signal to complete lip-sync and temporal alignment, and outputs an H.264 video stream for front-end playback.

[0063] The effectiveness of the present invention will be verified through experiments below: 1. Experimental conditions: The edge device uses two Phytium development boards (Phytium A and Phytium B), equipped with a Phytium quad-core processor and 4GB of memory; the cloud GPU uses an Intel 3090 with 32GB of video memory.

[0064] 2. Performance Testing: In this experiment, three architectural schemes were designed for comparison to evaluate the impact of different architectures on generation quality and inference efficiency in intelligent medical consultation scenarios: Cloud-only: Phytium A only handles data collection, directly uploading raw images and audio streams to the cloud server. The cloud then sequentially completes facial recognition, speech recognition, large-scale model inference, and finally generates a digital human video stream. The entire process relies entirely on cloud computing resources.

[0065] End-only: After Phytium A completes facial recognition and speech recognition, it sends the structured text information to Phytium B. The model deployed on B (approximately 1.5b) independently completes question-answering reasoning locally, and the generated results are directly handed over to the digital human engine to synthesize and play videos.

[0066] End-Cloud (this method): After Phytium A completes facial recognition and speech recognition, it sends the structured information to Phytium B. B deploys a small model (0.3b) to assist and accelerate the inference process of the large model (8b) in the cloud using the idea of ​​speculative sampling. The final inference result is synthesized into a video by the digital human engine and played.

[0067] It is worth noting that, due to the scale of the models, the digital human interaction models are uniformly deployed in the cloud, and the end devices realize the display of the digital human image and push stream by calling APIs and network interfaces.

[0068] In the experiment, this embodiment tested the inference latency of different architectures for the unified question: "What should I do if my eyes are uncomfortable?", as shown in Table 1. In the large model inference stage, this embodiment calculated the inference latency for every 100 tokens output.

[0069] Table 1. Staged inference latency for different architectures

[0070] Experimental results clearly demonstrate that the edge-cloud collaborative solution proposed in this embodiment significantly outperforms the end-only architecture in terms of inference latency, achieving a performance improvement of approximately 40.62%. This is primarily because deploying large language models purely on the edge is severely limited by device computing power, making it difficult to meet the requirements for low-latency response. Furthermore, this embodiment achieves approximately 2.5 times the speedup compared to the approach of decoding large edge models token by token. This significant improvement is attributed to the innovative speculative sampling mechanism, which allows the target model to process multiple tokens generated from the draft model at once, effectively reducing the overhead of serial computation and alleviating the memory bandwidth bottleneck.

[0071] While this invention does not offer an absolute advantage in inference latency compared to cloud-only architectures, it demonstrates unique value in scenarios that balance privacy protection and resource consumption. Although pure cloud solutions may achieve the lowest latency in certain metrics, uploading all data to the cloud may pose a risk of data privacy breaches and incur higher cloud computing costs and bandwidth consumption.

[0072] The edge-cloud collaborative architecture adopted in this invention ensures the localized processing of sensitive user data and effectively protects user privacy by deploying a lightweight model on the Phytium platform for initial processing and draft generation. Simultaneously, this architecture reduces reliance on cloud computing resources, effectively controlling operating costs. Through this collaborative approach, this invention significantly improves question-and-answer quality while guaranteeing data privacy and optimizing resource consumption. The powerful large-scale model in the cloud can verify and fine-tune the drafts generated on the edge, ensuring the professionalism and accuracy of the responses. Ultimately, this provides a more balanced and feasible intelligent medical assistant solution for practical applications.

[0073] This example also compares the response content of End-only and End-CIoud.

[0074] End-only: "Based on your description, eye discomfort may be caused by stress, fatigue, or other health factors. It is recommended to gradually reduce the strain on your eyes in daily life, such as reducing the time spent using electronic devices or engaging in appropriate relaxation activities. If symptoms persist or worsen, it is advisable to consult a doctor for professional guidance and diagnosis." End-CIoud: "As a man aged 20-28, if you experience eye discomfort, it could be due to a variety of reasons, including fatigue, dry eye, vision problems, or other health issues. Here are some suggestions to help relieve eye discomfort:" Give your eyes a rest: If you've been using electronic devices like computers, phones, or TVs a lot lately, consider stopping or at least reducing your screen time. Give your eyes a break.

[0075] Keep it clean: Make sure the skin around your eyes is clean, especially the eyelids and corners of your eyes. Use a gentle eye cleanser and a soft eye towel.

[0076] Moisturize your eyes: If your eyes feel dry, try using eye drops or eye ointments to keep them moist. You can also choose a moisturizing eye mask that contains eye drops.

[0077] Avoid overuse of eye drops: If you've already started using eye drops, avoid overuse. Overuse of eye drops may cause eye discomfort or irritation.

[0078] Check your glasses: If you wear glasses, make sure they fit you and there are no problems. Consider seeing an ophthalmologist to ensure your vision and eye health.

[0079] Seek professional help: If your eye discomfort persists for an extended period, or if you have any vision problems, it is recommended that you see an ophthalmologist for an examination. They can help diagnose the problem and provide appropriate treatment.

[0080] Maintaining a healthy lifestyle: Maintaining a healthy lifestyle, including a good diet, adequate sleep, and appropriate exercise, can help improve your overall health, including eye health.

[0081] If you experience any serious symptoms, such as severe pain, vision loss, or sudden eye problems, seek medical help immediately. The responses from End-only and End-Cloud models reveal significant limitations in the quality of output generated by the End-only architecture. Due to its smaller model size, the End-only model falls short in semantic understanding, content richness, and the personalization and professionalism of its suggestions, offering only brief and general recommendations that are insufficient for complex medical inquiries. In contrast, while the End-Cloud architecture proposed in this invention increases in inference latency, it leverages the powerful expressive and inference capabilities of a large cloud-based model to generate more comprehensive, targeted, and actionable results, making it particularly suitable for intelligent medical applications requiring high-quality output. Therefore, this invention achieves a trade-off between performance and quality, significantly improving answer quality while maintaining a certain response speed.

[0082] Table 2 Resource Usage of Different Architectures

[0083] In the experiment, this embodiment also evaluated the resource consumption of different architectures, as shown in Table 2. Experimental results show that compared to the End-only architecture, the present invention significantly reduces memory consumption by approximately 57.69%; compared to the Cloud-only architecture, it reduces GPU memory consumption by approximately 16.67%. This demonstrates that the on-device draft model preprocessing and speculative sampling mechanism effectively alleviate the inference burden of large cloud models. This optimized resource utilization allows the system to significantly reduce operating costs while maintaining high performance, making it more economical and feasible for practical deployment.

[0084] This embodiment also provides an intelligent medical question-and-answer system based on multimodal edge-cloud collaboration, which adopts the following technical solution: including edge devices and cloud devices.

[0085] The edge device is used to acquire visual images and speech data in real time; generate structured visual feature vectors based on visual images; generate speech-text feature vectors based on speech data; semantically align the visual feature vectors and speech-text feature vectors and fuse them into a unified format multimodal structured input package; send the input package to a draft model to generate multiple parallel candidate drafts; transmit the candidate drafts and their corresponding draft model probabilities, as well as the input package, to a cloud device; and display the digital human audio and video results; wherein, the draft model is a lightweight large language model.

[0086] The cloud device is used to verify and refine the candidate drafts using a multi-draft speculative sampling method based on importance weighting, generating high-quality question-and-answer text; inputting the high-quality question-and-answer text into the digital human engine to generate digital human audio-visual results; and pushing the digital human audio-visual results to the edge device. The target model is a high-performance large language model.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent medical question and answer method based on multi-modal end-cloud collaboration, characterized in that, Comprise: S1: real-time acquisition of visual images and voice data by an end-side device; S2: generating structured visual feature vectors from visual images; generating voice text feature vectors from voice data; S3: semantic alignment of visual feature vectors and voice text feature vectors, and fusion into a unified format of multi-modal structured input package; S4: inputting the input package into a draft model to generate multiple parallel candidate drafts; S5: transmitting the candidate drafts, their corresponding draft model probabilities, and the input package to a cloud-side device; the target model uses a multi-draft speculation sampling method based on importance weighting to verify and fine-tune the candidate drafts, generating high-quality question and answer text; S6: inputting the high-quality question and answer text into a digital human engine to generate digital human audio and video results; S7: pushing the digital human audio and video results to the end-side device for display.

2. The intelligent medical question and answer method based on multi-modal end-cloud collaboration of claim 1, wherein, The multi-draft speculation sampling method based on importance weighting is: according to the probability distribution of the candidate drafts under the draft model and the target model, the importance sampling algorithm is used to calculate the weight corresponding to each candidate draft, and the optimal intermediate draft is selected from the candidate drafts, wherein the draft model is a lightweight large language model, and the target model is a high-performance large language model; the intermediate draft is verified by speculation.

3. The intelligent medical question and answer method based on multi-modal end-cloud collaboration of claim 2, wherein, For K candidate drafts where, is the 1st candidate draft, is the Kth candidate draft; the optimal intermediate draft Conditional distribution is defined as: wherein, a sequence of K candidate drafts sampled from the draft model, the probability that a draft is selected as the optimal intermediate draft given K candidate drafts, the probability that a draft is selected as the optimal intermediate draft given K candidate drafts, For each , the following constraints are met: and wherein is a set of all possible candidate drafts; the selection probability of is: wherein, generating a draft for the target model the probability of, generating the i-th draft for the draft model the probability of, taking the minimum value, taking the maximum value; The maximization is performed over all that satisfy and: and: If then .

4. The intelligent medical question and answer method based on multi-modal end-cloud collaboration of claim 1, wherein, The visual image contains a face image; a lightweight visual recognition model deployed on the end-side device uses a convolutional neural network to extract gender and age label information from the visual image.

5. The intelligent medical question and answer method based on multi-modal end-cloud collaboration of claim 1, wherein, The voice data contains natural language inquiry content; a speech recognition model deployed on the end-side device uses a non-autoregressive speech recognition framework to generate vector representations containing semantics and emotions based on voice data.

6. The intelligent medical question and answer method based on multi-modal end-cloud collaboration of claim 1, wherein, The input package uses JSON format, containing timestamp, semantic label, user portrait identification and feature vector, and is encapsulated through lightweight encryption protocol.

7. The intelligent medical question and answer method based on multi-modal end-cloud collaboration of claim 1, wherein, The draft model is distilled and 4-bit quantized and compressed, and fine-tuned based on a medical domain pre-training dataset, In the training phase, its loss function is: wherein, is a conditional probability, is the i-th word in the generated sequence, is the 1-st to i-1-st words in the generated sequence, is a multi-modal input information, is a sequence length.

8. The intelligent medical question and answer method based on multi-modal end-cloud collaboration of claim 1, wherein, At each decoding step of the draft model, first truncate the output distribution by Top-K, then vectorize the truncated output distribution and perform parallel sampling and diversified beam search to obtain K candidate drafts.

9. The intelligent medical question and answer method based on multi-modal end-cloud collaboration of claim 1, wherein, The digital human engine is based on ER-NeRF technology.

10. An intelligent medical question and answer system based on multi-modal end-cloud collaboration, characterized in that, To perform a multi-modal end-cloud collaborative intelligent medical question and answer method as claimed in any one of claims 1 to 9, comprising: an end-side device and a cloud-side device, The end-side device is configured to: real-time acquisition of visual images and voice data; generating structured visual feature vectors from visual images; generating voice text feature vectors from voice data; semantic alignment of visual feature vectors and voice text feature vectors, and fusion into a unified format of multi-modal structured input package; inputting the input package into a draft model to generate multiple parallel candidate drafts; transmitting the candidate drafts, their corresponding draft model probabilities, and the input package to a cloud-side device; displaying the digital human audio and video results. The cloud device is configured to verify and fine-tune the candidate draft by using a target model and an importance-weighted multi-draft speculative sampling method to generate high-quality question and answer text; input the high-quality question and answer text into a digital human engine to generate digital human audio and video results; and push the digital human audio and video results to the terminal device.

Images