A wearable dermatosis AI-assisted diagnosis and treatment system, device and medium

Abstract

The application discloses a wearable skin disease AI auxiliary diagnosis and treatment system, equipment and medium, relates to the technical field of medical artificial intelligence, and comprises: wearable smart glasses are used for collecting skin lesion image data, doctor gaze behavior data and doctor-patient dialogue audio data, constitute multi-modal data, and real-time display skin diagnosis and treatment auxiliary information; a data processing and fusion platform is used for receiving the multi-modal data, pre-processing and feature extraction of the multi-modal data, outputting image features, gaze behavior features and text features; a multi-modal information fusion model is used for receiving the image features, gaze behavior features and text features, carrying out deep fusion and processing super-long multi-modal context through a multi-level fusion strategy based on an attention mechanism, and generating skin diagnosis and treatment auxiliary information; the application realizes deep semantic fusion and intelligent reasoning by collecting multi-modal diagnosis and treatment data in real time, combining a cloud multi-modal information fusion model, and providing diagnosis and treatment assistance for doctors.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, specifically to a wearable AI-assisted diagnosis and treatment system, device, and medium for skin diseases. Background Technology

[0002] Traditional dermatological diagnosis relies heavily on the doctor's visual observation, accumulated experience, and auxiliary tools such as dermoscopy. During diagnosis and treatment, doctors must simultaneously process information from multiple sources, such as observing the macroscopic morphology and microscopic structure of lesions, reviewing the patient's medical history, considering differential diagnoses, consulting medical literature, and communicating effectively with the patient. This multitasking can easily lead to cognitive overload, potentially causing the omission of key information, low diagnostic efficiency, or limited accuracy, especially for skin diseases with complex morphology, subtle early lesions, or those easily confused with similar conditions.

[0003] However, most existing assisted diagnostic systems are based on desktop computers or handheld devices, and their main functions focus on image analysis or information retrieval. They generally suffer from the following shortcomings: First, they cannot seamlessly integrate dynamically generated information such as images acquired by doctors during diagnosis and treatment, doctor observations, and doctor-patient dialogues, resulting in low information utilization efficiency and the formation of "information silos." Second, they lack the ability to understand the context of the doctor's actual diagnosis and treatment scenario; the information pushed by the system is often general rather than precisely tailored to the doctor's current focus and intent, reducing the effectiveness of the assistance. In doctor-patient communication, the system also cannot intelligently guide doctors to ask follow-up questions about key information. Third, traditional systems struggle to provide real-time, immersive assistance during the doctor's diagnosis and treatment process, requiring doctors to frequently switch attention, affecting the continuity of the treatment flow. Fourth, relying solely on visible light images makes it difficult to accurately assess deep pigmented or vascular lesions of the skin, such as deep melanomas or hemangiomas, limiting the ability for early detection and accurate diagnosis. Summary of the Invention

[0004] The purpose of this invention is to provide a wearable AI-assisted diagnosis and treatment system, device and medium for skin diseases, in order to solve the problems of low information integration, insufficient context understanding and difficulty in effectively handling ultra-long multimodal contexts in the existing technology for skin diagnosis.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] According to the first aspect of this disclosure, a wearable AI-assisted diagnosis and treatment system for skin diseases is proposed, comprising:

[0007] Wearable smart glasses are used to collect image data of skin lesions, doctor's gaze behavior data, and doctor-patient dialogue audio data to form multimodal data and display skin diagnosis and treatment auxiliary information in real time;

[0008] A data processing and fusion platform is used to receive the multimodal data, preprocess and extract features from the multimodal data, and output image features, gaze behavior features and text features;

[0009] A multimodal information fusion model is used to receive the image features, gaze behavior features, and text features, and to perform deep fusion and process ultra-long multimodal contexts through a multi-level fusion strategy based on an attention mechanism to generate auxiliary information for skin diagnosis and treatment; the auxiliary information for skin diagnosis and treatment includes reference diagnostic results, treatment plans, suggestions for follow-up questions, and risk warnings.

[0010] Furthermore, the data processing and fusion platform includes a multimodal data receiving module, a data preprocessing module, and a multimodal feature extraction module;

[0011] The multimodal data receiving module is also used to interface with a hospital information system or electronic medical record system to obtain patients' historical medical record data.

[0012] The multimodal feature extraction module includes an image feature extraction unit, a gaze behavior feature extraction unit, and a text feature extraction unit; the image feature extraction unit uses a residual network to extract the image features, and both the gaze behavior feature extraction unit and the text feature extraction unit use a Transformer module for feature encoding.

[0013] Furthermore, the multimodal information fusion model includes a multimodal fusion layer, a long context management module, and a diagnostic reasoning and knowledge recommendation module;

[0014] The multimodal fusion layer employs a multi-head cross-attention mechanism to perform mutual queries and information enhancement between the image features and the gaze behavior features, between the image features and the text features, and between the gaze behavior features and the text features.

[0015] Furthermore, the long context management module employs a sparse attention mechanism combined with an adaptive context window mechanism to dynamically adjust the size of the processing context window and capture long-distance dependencies during the diagnosis and treatment process.

[0016] Furthermore, the diagnostic reasoning and knowledge recommendation module generates the suggested follow-up questions by: identifying information gaps or diagnostic uncertainties based on the current diagnostic context and the collected image data, gaze behavior data, and text data, and dynamically generating targeted follow-up questions.

[0017] Furthermore, the wearable smart glasses are equipped with a high-resolution camera module, a gaze tracking module, an automatic optical magnification or zoom module, an audio acquisition module, and a display and interaction module;

[0018] The high-resolution camera module includes at least one visible light camera and one polarized light camera;

[0019] The gaze behavior tracking module has a built-in iris tracking sensor to obtain the doctor's gaze point, gaze duration, and gaze path;

[0020] The automatic optical zoom or digital zoom module is used to trigger optical zoom or digital zoom based on the gaze point and gaze duration.

[0021] The audio acquisition module has a built-in microphone array for acquiring audio of doctor-patient conversations;

[0022] The display and interaction module is equipped with an optical waveguide lens, which is used to display the skin diagnosis and treatment auxiliary information on the lens in real time.

[0023] Furthermore, the automatic optical magnification or scaling module is configured as follows:

[0024] When the system detects that a doctor's gaze on a certain area continues for more than a preset time threshold, the high-resolution camera module is automatically triggered to perform optical zoom or digital zoom to obtain a high-resolution local detail image of that area.

[0025] According to a second aspect of this disclosure, a wearable smart glasses is proposed for use as the data acquisition and interaction device in the first aspect, comprising:

[0026] A high-resolution camera module comprising at least one visible light camera and one polarized light camera;

[0027] gaze tracking module with built-in iris tracking sensor;

[0028] An automatic optical magnification or zoom module is electrically connected to the gaze behavior tracking module;

[0029] Audio acquisition module with built-in microphone array;

[0030] The display and interaction module is equipped with an optical waveguide lens for displaying skin diagnosis and treatment auxiliary information from the system.

[0031] According to a third aspect of this disclosure, a computer-readable storage medium is proposed, the medium storing computer instructions, which, when executed by a processor, are used to implement the functions of the data processing and fusion platform and the multimodal information fusion model described in the first aspect, wherein the computer-readable storage medium is a storage medium of a cloud GPU server.

[0032] According to the fourth aspect of this disclosure, a wearable AI-assisted diagnosis and treatment method for skin diseases is proposed, comprising the following steps:

[0033] S1. Image data of skin lesions, doctor's gaze behavior data, and doctor-patient dialogue voice data are collected through wearable smart glasses. The above three types of data together constitute multimodal data.

[0034] S2. Preprocess and extract features from the multimodal data to obtain image features, gaze behavior features, and text features;

[0035] S3. Input the image features, gaze behavior features and text features into the multimodal information fusion model, and generate a unified multimodal feature representation through multi-level fusion based on multi-head cross-attention and long context management;

[0036] S4. Based on the unified multimodal feature representation, perform intelligent reasoning to generate auxiliary information for skin diagnosis and treatment;

[0037] S5. The skin diagnosis and treatment auxiliary information is pushed to the doctor in real time through the display and interaction module of the wearable smart glasses.

[0038] Compared with existing technologies, this invention provides a wearable AI-assisted diagnosis and treatment system, device, and medium for skin diseases. It collects real-time visual, behavioral, and auditory multimodal data from the treatment site via wearable smart glasses, integrating patient medical history information. After preprocessing and feature extraction, all data is fed into a multimodal information fusion model. Through multi-level fusion based on attention mechanisms and long context management, a comprehensive diagnostic context representation is constructed. The multimodal information fusion model performs intelligent reasoning based on the fused features, generating personalized diagnostic assistance information, which is then fed back to the doctor in real-time via the wearable glasses, forming a closed-loop intelligent auxiliary decision-making process that significantly improves the accuracy, efficiency, and safety of skin diagnosis. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0040] Figure 1 This is a schematic diagram illustrating the overall architecture and data flow of the wearable AI-assisted diagnosis and treatment system for skin diseases of the present invention;

[0041] Figure 2 This is a schematic diagram of the overall architecture of the wearable AI-based intelligent medical system, device, and medium for dermatological diagnosis of the present invention.

[0042] Figure 3 This is a flowchart of the wearable AI-assisted diagnosis and treatment method for skin diseases according to the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] The wearable AI-assisted diagnosis and treatment system for skin diseases described in this invention includes wearable smart glasses, a data processing and fusion platform, and a multimodal information fusion model. The wearable smart glasses, as the front-end data acquisition and interaction terminal, are responsible for collecting images of skin lesions, doctor's gaze behavior, and audio of doctor-patient dialogue, and displaying diagnostic and treatment assistance information in real time. The data processing and fusion platform, as the middle layer, is responsible for receiving multimodal data and performing preprocessing and feature extraction. The multimodal information fusion model, deployed on a cloud GPU server, is responsible for deep fusion and intelligent reasoning of multimodal features to generate the final skin diagnosis and treatment assistance information.

[0045] Among them, wearable smart glasses integrate a high-resolution camera module, a gaze tracking module, an automatic optical magnification or zoom module, an audio acquisition module, and a display and interaction module;

[0046] High-resolution camera module: Includes at least one high-resolution visible light camera and one high-resolution polarized light camera. The visible light camera is used to capture routine images of skin lesions, providing macroscopic global information such as color, shape, and distribution; the polarized light camera, by eliminating reflections on the skin surface, obtains clearer information about subcutaneous structures, such as pigment networks, vascular textures, and hair follicle openings, to help distinguish between epidermal and dermal lesions;

[0047] Gaze behavior tracking module: Built-in high-precision iris tracking sensor and gaze tracking algorithm, which can monitor and record the doctor's gaze point, gaze duration and gaze path in real time, reflecting the doctor's visual focus and potential observation intention;

[0048] Automatic optical zoom or zoom module: Electrically connected to the gaze behavior tracking module, when it is detected that the doctor's gaze on a certain area continues for more than a preset time threshold (such as 2 seconds), the high-resolution camera module is automatically triggered to perform optical zoom or digital zoom to obtain a high-resolution local detail image of the area without the need for manual operation by the doctor.

[0049] Audio acquisition module: Built-in microphone array for high-fidelity acquisition of the conversation between doctors and patients, capturing information such as symptoms, medical history, and questions uttered during the diagnosis and treatment process;

[0050] Display and Interaction Module: Equipped with optical waveguide lenses, this module uses MicroLED optical waveguide technology to display images and text information within the doctor's field of vision, achieving non-invasive information display. It also supports multiple interaction methods such as voice commands, gesture recognition, and eye-tracking commands, facilitating convenient system operation for doctors during diagnosis and treatment.

[0051] The data processing and fusion platform includes a multimodal data receiving module, a data preprocessing module, and a multimodal feature extraction module;

[0052] The multimodal data receiving module can wirelessly connect with wearable smart glasses to receive collected image data, gaze behavior data, and audio data. It can also interface with hospital information systems (HIS) or electronic medical record systems (EMR) through standard interfaces to obtain patients' historical medical records.

[0053] The data preprocessing module is used to perform noise reduction, contrast enhancement, color correction, and image registration on image data; to decouple polarized light images to separate epidermal and dermal information; to perform smoothing filtering on gaze behavior data; and to convert audio data into text data using automatic speech recognition (ASR).

[0054] The multimodal feature extraction module includes an image feature extraction unit, a gaze behavior feature extraction unit, and a text feature extraction unit. The image feature extraction unit uses a ResNet residual network to extract macroscopic features from visible light images and microscopic features from polarized light images, respectively. The gaze behavior feature extraction unit uses a Transformer module to encode gaze point sequences and extract doctor observation pattern features. The text feature extraction unit uses a Transformer module to semantically encode doctor-patient dialogue text and patient medical record text, extracting key symptom and medical history features.

[0055] The multimodal information fusion model comprises a multimodal fusion layer, a long context management module, and a diagnostic reasoning and knowledge recommendation module. The multimodal fusion layer employs a multi-head cross-attention mechanism to enable mutual querying and information enhancement between image features and gaze behavior features, image features and text features, and gaze behavior features and text features. Through this mechanism, image features can focus on the lesion areas that doctors are actually concerned with, visual evidence can be closely associated with verbal symptoms, and doctors' gaze behavior can corroborate the dialogue content, thereby constructing a highly context-aware unified multimodal feature representation.

[0056] The long context management module employs a sparse attention mechanism combined with an adaptive context window mechanism. The sparse attention mechanism focuses on local context through sliding window attention, while combining global-local attention to capture long-distance dependencies, reducing computational complexity; the adaptive context window mechanism dynamically adjusts the size of the processing context window according to the complexity and information density of the diagnosis and treatment context, ensuring that more details can be captured at critical moments and maintaining high efficiency at non-critical moments;

[0057] The diagnostic reasoning and knowledge recommendation module uses a unified multimodal feature representation for intelligent reasoning to generate auxiliary information for skin diagnosis and treatment, including reference diagnostic results, treatment plans, suggested follow-up questions, and risk warnings. Specifically, the suggested follow-up questions dynamically generate targeted clinical questions based on the current diagnostic context and collected data, identifying information gaps or diagnostic uncertainties to guide doctors in collecting more comprehensive information.

[0058] Example

[0059] This embodiment aims to illustrate how the system of the present invention can assist doctors in the early identification and diagnosis of malignant melanoma in a real clinical setting.

[0060] 1. Data Acquisition and Intelligent Zoom:

[0061] Doctors wear wearable smart glasses to observe suspicious moles on patients' skin. A high-resolution visible light camera captures macroscopic images of the moles in real time, recording their location, size, shape, color unevenness, and edge features; a polarized light camera works simultaneously to acquire deep structural images of the moles, clearly displaying key microscopic features such as pigment reticulum, blue-white veil structure, and pseudopodia.

[0062] When a doctor makes an initial observation of a suspicious mole on a patient's skin, a high-resolution visible light camera in the glasses captures a macroscopic image of the mole in real time, clearly recording its location, overall size, shape, color unevenness, edge features, and approximate distribution on the body.

[0063] Simultaneously, a high-resolution polarized light camera in the glasses works to acquire a polarized light image of the mole. This polarized light image, by eliminating specular reflections on the skin surface, can more clearly reveal the structural information of the deeper layers of the skin. For example, in cases where melanoma is suspected, it can more clearly display key microscopic features such as pigment networks, blue-white veil structures, pseudopodia, as well as vascular patterns or clusters of pigment cells in the dermis.

[0064] The gaze tracking module uses a high-precision iris tracking sensor to accurately monitor the doctor's eye movements. For example, when the system detects that the doctor is focusing on a specific area of ​​the mole (such as an area with irregular borders or abnormal color) for an extended period (e.g., continuously staring for more than a preset 2-second threshold) or repeatedly, the system identifies the doctor's high level of attention and potential questions about this area. At this time, the automatic optical magnification / zoom module responds immediately, intelligently driving the camera to perform optical zoom or high-magnification digital zoom, magnifying the lesion area currently being looked at by the doctor (such as an irregular border), and automatically acquiring a more detailed, high-resolution image of the local details. This process is fully automated, requiring no manual operation from the doctor, ensuring the continuity of the diagnosis and treatment process and the immediate acquisition of detailed information.

[0065] The microphone array of the audio acquisition module continuously records the conversation between the doctor and the patient throughout the entire treatment process. For example, the doctor asks, "When was this mole discovered? Has it recently increased in size, darkened, itched, or bled?"; the patient replies, "It seems to have gotten a little bigger recently, and it itches occasionally."

[0066] 2. Multimodal data preprocessing and feature extraction:

[0067] Image preprocessing: The acquired macroscopic visible light images and microscopic polarized light images are first subjected to image preprocessing algorithms for denoising, contrast enhancement, color correction, and image registration to ensure consistent image quality and corresponding skin lesions at different scaling sizes. For polarized light images, decoupling processing is also performed to separate the independent information of the epidermis and dermis. Subsequently, image registration is performed to ensure accurate alignment of images acquired by polarized and unpolarized cameras.

[0068] Image Feature Extraction: Preprocessed images are uploaded to a cloud server in real time. Macroscopic and microscopic images are input into two deep residual networks (ResNet-152) pre-trained on large-scale dermatology datasets to extract macroscopic and microscopic visual features of the lesions, respectively. Macroscopic image features include overall morphology, edge sharpness, color uniformity, symmetry, and distribution patterns on the body. Microscopic image features are extracted, focusing on more discriminative microstructural features such as the integrity and uniformity of the pigment network, the presence of atypical blood vessels, and the presence of blue-white structures or homogeneous regions.

[0069] Fixation behavior feature extraction: The original fixation point sequence (x, y coordinates and timestamps) is smoothed and filtered before being input into the Transformer encoder. This network learns the dynamic patterns of doctors observing lesions, extracts high-level fixation features, and calculates the proportion of fixation time spent in the lesion periphery, the distribution entropy of fixation points within the lesion, the fixation point jump frequency, and the fixation pattern deviation compared to normal skin. These features reflect the doctor's observational intentions, focus of attention, and concerns about potential abnormalities in the lesion.

[0070] Text Feature Extraction: The audio of the doctor-patient dialogue is converted into text by a high-performance automatic speech recognition (ASR) module. All obtained text information (including the dialogue content from ASR and patient medical record text obtained from HIS / EMR, such as age, gender, family history, past medical history, etc.) is input into a pre-trained natural language processing model to generate context-sensitive semantic embeddings, capturing key symptom descriptions (such as "itching" and "enlargement"), medical history information (such as "family history"), and emotions or questions that may be implied in the patient's self-report.

[0071] 3. Multimodal information fusion and diagnostic reasoning:

[0072] All extracted image features, gaze behavior features, and text features are fed into the multimodal information fusion layer.

[0073] This fusion layer employs a deep, multi-level fusion strategy with a multi-head cross-attention mechanism. This mechanism allows feature vectors from different modalities to perform mutual queries, key-value matching, and information augmentation within the deep network, thereby achieving a comprehensive understanding of the diagnostic context.

[0074] Image-Gaze ​​Cross-Attention: Image features (as queries) can actively query gaze behavior features (as keys and values), enabling the final image representation to "focus" on the local lesion area that the doctor is actually focusing on, effectively filtering out background noise and interference from irrelevant areas, thereby improving the representativeness of lesion features. Conversely, gaze behavior features can also query image features to gain a deeper understanding of the specific visual content and subtle features of the doctor's gaze.

[0075] Image-text cross-attention: Enables interaction between image features and dialogue and medical record text features. For example, when an image shows a pigmented spot with irregular edges and uneven color, and the doctor-patient dialogue mentions "recent enlargement" and "occasional itching," the model can closely associate these visual evidences with the verbal symptoms, significantly enhancing the diagnostic weight for specific diseases (such as malignant melanoma and dysplastic nevus).

[0076] Gaze-text cross-attention: Combining a doctor's gaze behavior (such as looking at a patient's face rather than the lesion) with the content of the conversation (such as asking about the patient's emotions and lifestyle habits) helps the model understand the doctor's non-diagnostic concerns, such as providing emotional comfort, verifying information, or obtaining information about lifestyle habits.

[0077] Through iterations of multiple cross-attention levels, the model can construct a highly context-aware, semantically rich unified multimodal joint feature representation, which can comprehensively capture visual cues of skin lesions, doctors' observational intentions, and relevant clinical background information.

[0078] Long Context Management: The fusion layer also integrates a long context management module to handle the continuous accumulation of data streams during diagnosis and treatment. This module employs a sparse attention mechanism: to reduce the secondary complexity of attention computation for ultra-long sequences, the model adopts a sparse attention pattern. By using a sliding window attention mechanism, it focuses only on local context, while combining global-local attention, allowing some attention heads to focus on distant contextual information or predefined global features, thereby capturing long-distance dependencies while maintaining computational efficiency. Simultaneously, it incorporates an adaptive context window mechanism: the model dynamically adjusts the size of its processing context window based on the complexity and information density of the diagnosis and treatment context. For example, in the initial observation stage, the context window is small; while when performing detailed differential diagnosis for complex lesions, the context window automatically expands to capture more historical images, dialogue, and behavioral information, ensuring that more details are captured at critical moments, while maintaining efficiency in non-critical moments.

[0079] The fused multimodal features (including long contextual information) are input into the diagnostic reasoning and knowledge recommendation module. This module includes:

[0080] Differential diagnosis prediction and ranking: A Softmax classifier is used to predict a list of the most likely skin disease diagnoses and their confidence levels based on fused features, along with supporting evidence (such as highlighted key image features and relevant dialogue fragments). For example, the output might be "Malignant melanoma (75% confidence), Atypical hyperplasia (20% confidence)".

[0081] Intelligent treatment recommendations: Combining diagnostic results and the patient's specific circumstances (such as allergy history and medication history) with the latest clinical guidelines obtained from the medical knowledge base, personalized treatment recommendations are made, including drug selection, surgical indications, physical therapy, etc.

[0082] Relevant medical knowledge retrieval and push: Based on the current diagnosis and treatment context (lesion characteristics, inferred diagnosis, doctor's gaze point, dialogue keywords), intelligently retrieve and push relevant medical literature abstracts, clinical guideline entries, similar case images or the latest research progress for doctors' reference.

[0083] Intelligent follow-up question generation: Based on the current diagnostic bias and collected information, the model uses natural language generation technology to identify potential information gaps or ambiguities in the diagnosis and treatment process. For example, if the image and known medical history strongly suggest malignant melanoma, but there is a lack of clear information about the "progression rate" of the lesion, the model will dynamically generate targeted, clinically valuable follow-up questions, such as: "When did this mole first appear? Have you noticed any significant increase in size or color change recently?" These questions aim to proactively guide doctors to collect more comprehensive and accurate information, assist in differential diagnosis, and improve consultation efficiency.

[0084] Potential risk identification and early warning: The model analyzes medical information in real time. When it identifies potential diagnostic biases (such as a large difference between AI predictions and the doctor's current inclination, or certain key features not being fully considered), medication contraindications (such as drugs that conflict with the patient's allergy history), or omissions that may occur during the diagnosis and treatment process (such as failure to examine swollen lymph nodes near the lesion), it promptly provides high-priority and clear early warning prompts through the glasses display screen to avoid medical risks.

[0085] 4. Information push and interaction:

[0086] The system uses the display module of smart glasses to display concise text on the doctor's lenses, pushing the aforementioned diagnostic suggestions, knowledge information, suggestions for follow-up questions, warning prompts, and key feature annotations to the doctor in real time and seamlessly.

[0087] The information push has a priority management mechanism to ensure that the most urgent and relevant information can be perceived by doctors first, while allowing doctors to filter or expand detailed information as needed.

[0088] Doctors can interact flexibly with the system through voice commands (such as “show differential diagnosis”, “zoom in on this area”, “record medical orders”, “next question”), simple gesture recognition (such as pinch to confirm, swipe to turn pages) or eye-tracking commands (such as staring at a specific icon to trigger a function), to retrieve more detailed information, switch views, skip follow-up questions, or enter a do-not-disturb mode, ensuring ease of operation and a smooth diagnostic and treatment process.

[0089] This embodiment clearly demonstrates how the system of the present invention provides precise and real-time intelligent assistance to doctors in actual clinical scenarios through multimodal data fusion, especially by utilizing polarized light images, doctor gaze behavior, and intelligent follow-up question generation, combined with a long context management module, thereby improving the early diagnosis rate and information collection efficiency of malignant melanoma, and has important clinical value.

[0090] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A wearable AI-assisted diagnostic and treatment system for skin diseases, characterized in that, include: Wearable smart glasses are used to collect image data of skin lesions, doctor's gaze behavior data, and audio data of doctor-patient dialogue to form multimodal data and display skin diagnosis and treatment auxiliary information in real time; A data processing and fusion platform is used to receive the multimodal data, preprocess and extract features from the multimodal data, and output image features, gaze behavior features and text features; A multimodal information fusion model is used to receive the image features, gaze behavior features, and text features, and to perform deep fusion and process ultra-long multimodal contexts through a multi-level fusion strategy based on an attention mechanism to generate auxiliary information for skin diagnosis and treatment; the auxiliary information for skin diagnosis and treatment includes reference diagnostic results, treatment plans, suggestions for follow-up questions, and risk warnings.

2. The wearable AI-assisted diagnosis and treatment system for skin diseases according to claim 1, characterized in that, The data processing and fusion platform includes a multimodal data receiving module, a data preprocessing module, and a multimodal feature extraction module; The multimodal data receiving module is also used to interface with a hospital information system or electronic medical record system to obtain patients' historical medical record data. The multimodal feature extraction module includes an image feature extraction unit, a gaze behavior feature extraction unit, and a text feature extraction unit; the image feature extraction unit uses a residual network to extract the image features, and both the gaze behavior feature extraction unit and the text feature extraction unit use a Transformer module for feature encoding.

3. The wearable AI-assisted diagnosis and treatment system for skin diseases according to claim 1, characterized in that, The multimodal information fusion model includes a multimodal fusion layer, a long context management module, and a diagnostic reasoning and knowledge recommendation module; The multimodal fusion layer employs a multi-head cross-attention mechanism to perform mutual queries and information enhancement between the image features and the gaze behavior features, between the image features and the text features, and between the gaze behavior features and the text features.

4. The wearable AI-assisted diagnosis and treatment system for skin diseases according to claim 3, characterized in that, The long context management module employs a sparse attention mechanism combined with an adaptive context window mechanism to dynamically adjust the size of the processing context window and capture long-distance dependencies during the diagnosis and treatment process.

5. A wearable AI-assisted diagnosis and treatment system for skin diseases according to claim 3, characterized in that, The diagnostic reasoning and knowledge recommendation module generates the suggested follow-up questions by: identifying information gaps or diagnostic uncertainties based on the current diagnostic context and the collected image data, gaze behavior data, and text data, and dynamically generating targeted follow-up questions.

6. The wearable AI-assisted diagnosis and treatment system for skin diseases according to claim 1, characterized in that, The wearable smart glasses are equipped with a high-resolution camera module, a gaze tracking module, an automatic optical magnification or zoom module, an audio acquisition module, and a display and interaction module. The high-resolution camera module includes at least one visible light camera and one polarized light camera; The gaze behavior tracking module has a built-in iris tracking sensor to obtain the doctor's gaze point, gaze duration, and gaze path; The automatic optical zoom or digital zoom module is used to trigger optical zoom or digital zoom based on the gaze point and gaze duration. The audio acquisition module has a built-in microphone array for acquiring audio of doctor-patient conversations; The display and interaction module is equipped with an optical waveguide lens, which is used to display the skin diagnosis and treatment auxiliary information on the lens in real time.

7. A wearable AI-assisted diagnosis and treatment system for skin diseases according to claim 6, characterized in that, The automatic optical magnification or scaling module is configured as follows: When the system detects that a doctor's gaze on a certain area continues for more than a preset time threshold, the high-resolution camera module is automatically triggered to perform optical zoom or digital zoom to obtain a high-resolution local detail image of that area.

8. A wearable smart glasses device, used as a data acquisition and interaction device for the wearable AI-assisted diagnosis and treatment system for skin diseases according to any one of claims 1 to 7, characterized in that, include: A high-resolution camera module comprising at least one visible light camera and one polarized light camera; gaze tracking module with built-in iris tracking sensor; An automatic optical magnification or zoom module is electrically connected to the gaze behavior tracking module; Audio acquisition module with built-in microphone array; The display and interaction module is equipped with an optical waveguide lens for displaying skin diagnosis and treatment auxiliary information from the system.

9. A computer-readable storage medium, characterized in that, The medium stores computer instructions, which, when executed by a processor, are used to implement the functions of the data processing and fusion platform and the multimodal information fusion model in the wearable skin disease AI-assisted diagnosis and treatment system as described in any one of claims 1 to 7. The computer-readable storage medium is the storage medium of a cloud GPU server.

10. A wearable AI-assisted diagnosis and treatment method for skin diseases, characterized in that, Includes the following steps: S1. Image data of skin lesions, doctor's gaze behavior data, and doctor-patient dialogue voice data are collected through wearable smart glasses. The above three types of data together constitute multimodal data. S2. Preprocess and extract features from the multimodal data to obtain image features, gaze behavior features, and text features; S3. Input the image features, gaze behavior features and text features into the multimodal information fusion model, and generate a unified multimodal feature representation through multi-level fusion based on multi-head cross-attention and long context management; S4. Based on the unified multimodal feature representation, perform intelligent reasoning to generate auxiliary information for skin diagnosis and treatment; S5. The skin diagnosis and treatment auxiliary information is pushed to the doctor in real time through the display and interaction module of the wearable smart glasses.

Images