Barrier-free intelligent doctor guiding method and platform based on multi-modal interaction

Abstract

The application provides a barrier-free intelligent doctor guiding method and platform based on multi-modal interaction, relates to the technical field of multi-modal interaction, and comprises the following steps: acquiring user time sequence state data according to a bimodal state acquisition unit to perform doctor guiding support state detection and output a target doctor guiding support mode; calling a target mode doctor guiding interaction container; using doctor guiding demand data input by a user to traverse a medical resource navigation graph to position an initial interaction node; converting an interaction response reply into a modal inquiry response to reply to the initial demand intention of the user; performing multi-round demand intention analysis according to the updated feedback demand of the user and outputting a department navigation suggestion label set; and packing the demand interaction record and the department navigation suggestion label set into a doctor guiding demand alarm and sending the doctor guiding demand alarm to a target auxiliary doctor guiding. The application solves the technical problem that the prior art usually relies on single mode interaction and cannot provide comprehensive support for special groups, thereby affecting user experience and service efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal interaction, in particular to an accessible intelligent doctor guiding method and platform based on multi-modal interaction. BACKGROUND

[0002] Traditional manual doctor guiding methods often cannot provide accurate and efficient services, especially for special groups such as the elderly and the disabled, so intelligent doctor guiding systems have emerged as the times require and have become an innovative technology for improving the medical experience of special groups and optimizing the scheduling of medical resources. Intelligent doctor guiding systems are used to help patients make efficient department selection and treatment path planning. However, existing intelligent doctor guiding systems often rely on a single interaction mode, such as voice or image, which makes it difficult to provide perfect services to users with visual or hearing impairments. For example, voice recognition systems cannot effectively serve the hearing impaired, and image recognition systems cannot help the visually impaired, which makes it difficult for these special groups to complete the medical process smoothly, affecting user experience and service efficiency. SUMMARY

[0003] The present application provides an accessible intelligent doctor guiding method and platform based on multi-modal interaction, aiming to solve the technical problem that existing technologies usually rely on a single mode of interaction and cannot provide comprehensive support for special groups, thereby affecting user experience and service efficiency.

[0004] The first aspect of the present application provides an accessible intelligent doctor guiding method based on multi-modal interaction, which comprises: acquiring user time sequence state data according to a dual-modal state acquisition unit to detect the guiding support state and output a target guiding support mode; calling a target modal guiding interaction container in a multi-modal guiding interaction platform according to the target guiding support mode; the target modal guiding interaction container uses user input guiding demand data to traverse a medical resource navigation graph to locate an initial interaction node; after extracting the interaction response reply of the initial interaction node, the interaction response reply is converted into a modal inquiry response according to the target guiding support mode to reply to the initial demand intention of the user; according to the updated feedback demand of the user to the modal inquiry response, a multi-round demand intention analysis is performed along the medical resource navigation graph, and a department navigation suggestion label set is output; after matching a target auxiliary doctor guiding according to the target guiding support mode, the demand interaction record and the department navigation suggestion label set are packaged as a doctor guiding demand alarm and sent to the target auxiliary doctor guiding.

[0005] In a second aspect, the application discloses an intelligent barrier-free doctor guiding platform based on multi-modal interaction, which is used for the intelligent barrier-free doctor guiding method based on multi-modal interaction, and comprises a state detection module, a container calling module, an interaction node positioning module, an intention reply module, an intention analysis module and an alarm sending module.

[0006] The one or more technical solutions provided in the application have at least the following beneficial effects:

[0007] The time sequence state data of the user is collected in real time by the dual-mode state collection unit, and the doctor guiding support state detection is automatically performed based on the data, so that the demand type of the user can be intelligently judged, and the most suitable doctor guiding support mode can be selected according to the physical condition and interaction preference of the user, thereby improving the individualization and intelligent level of the medical experience; by calling the corresponding target mode doctor guiding interaction container according to the target doctor guiding support mode, the user can be provided with an interactive mode suitable for his / her needs, and personalized interactive modes can be provided for different user groups, thereby significantly improving the adaptability of the intelligent doctor guiding system to special groups; by traversing the medical resource navigation graph based on the doctor guiding demand data input by the user, the initial interaction node is located, and by using the intelligent medical resource navigation graph, the most suitable department recommendation can be quickly responded and provided, thereby reducing the navigation time and unnecessary consultation of the user in the hospital and improving the optimization scheduling and use efficiency of the hospital resources; after extracting the interaction response reply of the initial interaction node, the interaction response reply is converted into a modal inquiry response meeting the needs of the user according to the target doctor guiding support mode, and this intelligent adaptation enhances the interactivity and user experience of the system, especially in the user groups with obstacles or special needs, the self-help ability of the users can be maximized and the waiting and information acquisition time can be reduced; according to the updated feedback needs of the user, multi-round demand intention analysis is performed along the medical resource navigation graph, the department navigation suggestion label is accurately optimized, and by continuously iterating the analysis, the recommended department can be dynamically adjusted, especially for users with changing needs or complex symptoms, the recommended path can be continuously optimized, and the possibility of recommendation errors or resource waste can be effectively reduced; by packaging the demand interaction record and the department navigation suggestion label set, they are sent to the target auxiliary doctor, and the target auxiliary doctor makes a decision and performs guiding service according to the data, this mechanism provides accurate data support to help the target auxiliary doctor make a quick decision and optimize the department scheduling and user guidance, especially in the case of resource shortage, the utilization efficiency of medical resources can be effectively improved and the waiting time of the user can be reduced.

[0008] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the present application can be implemented in accordance with the content of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 The flowchart of the barrier-free intelligent doctor guiding method based on multi-modal interaction provided by the embodiments of the present application.

[0010] Figure 2 The structure diagram of the barrier-free intelligent doctor guiding platform based on multi-modal interaction provided by the embodiments of the present application.

[0011] Legend: state detection module 10, container calling module 20, interactive node positioning module 30, intention reply module 40, intention analysis module 50, and alarm sending module 60. DETAILED DESCRIPTION

[0012] The embodiments of the present application provide a barrier-free intelligent doctor guiding method and platform based on multi-modal interaction, which solves the technical problem that the prior art usually relies on single-modal interaction and cannot provide comprehensive support for special groups, thereby affecting user experience and service efficiency.

[0013] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be specifically introduced in combination with the drawings of the specification. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0014] Embodiment one, as shown in the figure, the embodiments of the present application provide a barrier-free intelligent doctor guiding method based on multi-modal interaction, the method comprises: Figure 1

[0015] The user time sequence state data obtained by the dual-modal state acquisition unit is used for guiding support state detection, and a target doctor guiding support mode is output.

[0016] The dual-modal state acquisition unit is used for simultaneously acquiring visual information and auditory information of the user. Specifically, the dual-modal state acquisition unit acquires image data of the user through a camera or an image sensor as visual information, including body movements, facial expressions, posture changes, etc. of the user, and acquires audio data of the user through a microphone or the like as auditory information, including the user's voice, speech, etc. The user time sequence state data refers to continuous data of the user in a period of time obtained by the dual-modal state acquisition unit, which is recorded in time sequence.

[0017] A plurality of detection methods are used to analyze the user time sequence state data, for example, human body pose key point detection is performed on the image by using computer vision technology to identify the posture and action of the user; the audio signal is extracted to identify the speech content, tone, rhythm, etc. of the user; optical flow analysis is performed on the body exploration action to identify whether the user explores or interacts with the environment through body action, such as whether to stretch out the hand to touch the object; feature detection of auxiliary equipment is performed to identify whether the user uses auxiliary equipment such as a wheelchair, a walking stick, etc.

[0018] The above detection results are integrated to determine the demand state of the user, and a barrier-free support mode is output. The barrier-free support mode represents the doctor guiding support mode required by the user in the current state, for example, if it is detected that the user has special needs such as hearing impairment, visual impairment, etc., then the barrier-free doctor guiding service mode is provided accordingly.

[0019] ​According to the target doctor-guiding support mode, a target mode doctor-guiding interaction container is called on the multimodal doctor-guiding interaction platform.

[0020] The multimodal doctor-guiding interaction platform supports multiple interaction modes such as vision, hearing, touch, and text. The multimodal doctor-guiding interaction platform determines the interaction mode with the user according to the target doctor-guiding support mode, and calls the corresponding target mode doctor-guiding interaction container. The target mode doctor-guiding interaction container is a module that can provide corresponding interaction functions according to different target doctor-guiding support modes. For example, if the target doctor-guiding support mode is voice, a voice dialogue container is called for voice guidance and interaction; if the target doctor-guiding support mode is vision, an image or visual interface container is called to provide image guidance or visual identification; if it is a hybrid mode, both voice and image containers are called to achieve a more intuitive and diversified guiding experience. Through the calling of the target mode doctor-guiding interaction container, personalized and barrier-free experience in the doctor-guiding process is ensured.

[0021] The target mode doctor-guiding interaction container uses the doctor-guiding demand data input by the user to traverse the medical resource navigation graph to locate the initial interaction node.

[0022] The user can input his / her doctor-guiding demand through multiple modes, for example, through voice input, or through gestures, text, etc. The system receives these input data and converts them into structured doctor-guiding demand data, which includes the user's symptom description, medical problem, and demand for a medical department, etc. The medical resource navigation graph is a knowledge graph that contains information about medical resources, departments, doctors, and diagnosis and treatment paths, etc. It is a tree graph, and the end layer nodes are specific department nodes, which help to locate the required medical service resources of the user.

[0023] In this step, the target mode doctor-guiding interaction container traverses the medical resource navigation graph according to the user's doctor-guiding demand data, matches the user's demand with the nodes in the graph according to the association rules in the medical resource navigation graph, and locates the most relevant initial interaction node in this way. The initial interaction node does not directly correspond to a department, but one or more inquiry nodes in the medical resource navigation graph. These inquiry nodes are guiding questions generated by the system according to the user's inputted preliminary symptoms or demand, similar to the question and answer interaction between the artificial doctor-guiding desk staff and the user. For example, if the user inputs headache as a symptom, the medical resource navigation graph is traversed according to the headache, and the inquiry nodes in the graph that match the keywords of the symptom description are matched, such as the question "Does your headache come with nausea or vomiting?" etc., as the initial interaction node.

[0024] After extracting the interaction response reply of the initial interaction node, the interaction response reply is converted into a modal inquiry response according to the target doctor-guiding support mode to reply to the user's initial demand intention.

[0025] extracting a corresponding interaction response from the interaction response reply library of the initial interaction node, the interaction response being a fixed answer, guidance information, suggestion or inquiry provided to the user, aiming to further confirm the user's needs or answer the user's questions, for example, if the user describes a headache, an initial interaction response will be extracted, asking "Does your headache have nausea or vomiting?"

[0026] The target doctor guidance support mode refers to the previously determined interaction mode with the user. According to the target doctor guidance support mode, the extracted interaction response is converted into a format matching the target doctor guidance support mode, ensuring that the user can receive information in the most suitable way, for example, if the target doctor guidance support mode is voice, the user will be played in the form of voice; if the target doctor guidance support mode is image, it will be presented in the form of chart, text or image; if it is a mixed mode, both voice and image will be used to feedback to the user.

[0027] After converting the interaction response into a suitable mode, it is output as a modal inquiry response. The modal inquiry response is not only a simple reply, but also includes further guidance or explanation of the user's initial needs.

[0028] According to the user's updated feedback needs for the modal inquiry response, multiple rounds of demand intention analysis are performed along the medical resource navigation graph, and a department navigation suggestion label set is output.

[0029] After receiving the modal inquiry response, the user reacts, such as further asking questions, confirming symptoms or expressing misunderstanding, generating updated feedback needs, for example, through voice feedback to express preference for a certain option or ask new questions; through text feedback to further input symptoms or description, or through menu selection to respond.

[0030] After analyzing the updated feedback needs, multiple rounds of demand intention analysis are performed along the medical resource navigation graph. Each user feedback helps the system further refine the needs, for example, the user initially says headache, and then supplements that it is accompanied by vomiting. According to this new information, further analysis of possible diseases is carried out, and through various nodes in the graph, such as departments, symptoms, doctors, etc., relevant departments or medical resources are gradually locked.

[0031] Based on the results of multiple rounds of demand intention analysis, a department navigation suggestion label set is generated, which is a set of recommended department or medical resource labels, such as internal medicine, neurology, gastroenterology, etc. Through the labels, artificial doctor guidance is provided with reference to help them quickly filter the most suitable department.

[0032] After matching the target auxiliary doctor guidance according to the target doctor guidance support mode, the demand interaction record and the department navigation suggestion label set are packaged as a doctor guidance demand alarm and sent to the target auxiliary doctor.

[0033] According to the target doctor guiding support mode matching target auxiliary doctor guiding, the target auxiliary doctor guiding is the staff of the hospital, responsible for receiving the doctor guiding demand and helping the user to select the appropriate department or medical resource. The demand interaction record includes the initial demand of the user, the subsequent feedback and the update information, and the demand interaction record and the department navigation suggestion label set are packaged as the doctor guiding demand alarm, which is sent to the target auxiliary doctor guiding through the hospital management system, the doctor guiding platform and the like. The target auxiliary doctor guiding confirms the target department based on the department navigation suggestion label set and the demand interaction record of the user, and performs the guiding service such as guiding the user to a specific department, arranging a doctor, guiding treatment and the like.

[0034] Further, according to the user time sequence state data obtained by the bimodal state acquisition unit, the doctor guiding support state detection is performed, and the target doctor guiding support mode is output. The method comprises the following steps:

[0035] When the user stays in the doctor guiding island platform monitoring area for a duration that meets a preset threshold, the bimodal state acquisition unit is activated to obtain the user time sequence state data; the time sequence state data is analyzed to obtain a time sequence image array and time sequence audio information; the doctor guiding support state detection is performed by using the time sequence image array and the time sequence audio information, and a barrier-free support mode is output; the target doctor guiding support mode is obtained by screening through the doctor guiding interaction response test performed on the user by using the interaction protocol corresponding to the barrier-free support mode.

[0036] The doctor guiding island platform monitoring area is a specific area, in which the activities of the user will be monitored and tracked. When the user enters the doctor guiding island platform monitoring area, the stay time of the user in the doctor guiding island platform monitoring area is tracked. If the stay time of the user exceeds a preset threshold, for example, several minutes, it is considered that the user needs doctor guiding support, and the bimodal state acquisition unit is activated at this time. The bimodal state acquisition unit comprises an image acquisition device (such as a camera) and an audio acquisition device (such as a microphone), which can simultaneously capture real-time image and sound data, and obtain user time sequence state data, providing basic information for subsequent state detection.

[0037] The time sequence state data is analyzed to obtain a time sequence image array and time sequence audio information. The time sequence image array refers to a sequence of image data captured continuously within a specific time period, which includes facial expressions, body movements, position changes and the like of the user. The time sequence audio information refers to a sequence of audio data recorded within the same time period, which includes the voice of the user, environmental sound and the like.

[0038] According to the time sequence image array and the time sequence audio information, the needs and state of the user are determined, wherein the time sequence image array helps to identify whether the user shows unusual behavior, such as anxiety, confusion, and the need for help; the audio data helps to identify whether the user has language barriers, voice expression difficulties, or rapid speech symptoms. The barrier-free support mode is the most appropriate interaction mode output according to the detected user needs, which supports multiple interaction modes, such as voice and text interaction modes for language barrier support. For example, if the user's behavior shows language barriers, voice assistance is used in combination with text instructions to help the user understand; if the user's hearing impairment or unclear speech is detected, text is output and visual guidance is used to guide the user.

[0039] The interaction protocol refers to the rules for effective communication between the system and the user for different barrier-free support modes. For example, if the user uses the voice mode, the system interacts with the user through voice recognition technology and gives feedback using appropriate speech synthesis technology; if it is a text mode, a graphical user interface is designed to provide image, text, and logo information for guidance.

[0040] According to the interaction protocol, the effectiveness of different interaction modes is tested by simulating interaction with the user, for example, using voice or text mode to test user feedback and see how well they accept information. The goal of the test is to determine the response of the user to each support mode and evaluate its effectiveness and accuracy.

[0041] Based on the results of the interaction response test, the most appropriate target doctor support mode is selected, for example, if the user shows good understanding of the text guidance, the text mode is finally selected as the main interaction mode with the user, and if the user can successfully pass the voice interaction for doctor guidance, the voice mode is finally selected as the main interaction mode.

[0042] Further, the method further comprises:

[0043] locally calling multiple sets of single-modal doctor guidance dialogue sequences, wherein the single-modal doctor guidance dialogue sequences are desensitized; performing cross-modal semantic alignment on the multiple sets of single-modal doctor guidance dialogue sequences to generate multiple doctor coding time sequence feature vectors; after key word analysis of the multiple doctor coding time sequence feature vectors, combining a symptom department association rule base to generate multiple structured intent sequences; performing intent clustering on the multiple structured intent sequences to obtain P standardized intent nodes; mapping the multiple structured intent sequences according to the P standardized intent nodes to construct the medical resource navigation graph, wherein the last layer nodes of the medical resource navigation graph are multiple department resource nodes.

[0044] The single-modal doctor-guiding dialogue sequence is a series of standardized and preset question-and-answer dialogues provided for users according to different interaction modes. These dialogues are defined by the system in advance, aiming to guide users to provide more information or answer their needs. Desensitization refers to the process of deleting or replacing sensitive information such as personal information, medical history, etc. in these single-modal doctor-guiding dialogue sequences to protect users' privacy and data security. For example, when calling a dialogue sequence, automatically remove users' personal names, phone numbers, or other sensitive information to ensure that dialogue data does not expose users' privacy.

[0045] Cross-modal semantic alignment refers to aligning information from different modalities so that the system can understand the semantic relationship between these modalities. In this case, multiple sets of single-modal doctor-guiding dialogues contain dialogue content in multiple different modalities such as voice and text. Through cross-modal semantic alignment, these different modalities of information can be mapped to a unified semantic space, thereby extracting their consistency at the semantic level. For example, aligning "I have a headache" described by the user in voice with "headache" represented in text as the same symptom expression. This way, the system can effectively process content in different input modalities. Doctor-guiding coding time-series feature vectors are time-series features extracted from multiple sets of single-modal doctor-guiding dialogues. These features represent the user's intentions, needs, symptoms, and other information expressed during the entire conversation. Since the user's conversation is time-sequential, these features are arranged in chronological order to form multiple doctor-guiding coding time-series feature vectors.

[0046] Chief complaint and demand keyword analysis refers to extracting the most core and important demand information from multiple doctor-guiding coding time-series feature vectors. Through dialogue analysis, the user's chief complaint, i.e., the user's most urgent medical demand, is identified. Chief complaint and demand keywords include symptoms, disease names, patient expectations or demands, etc. The symptom department association rule base is a knowledge base containing the relationship between symptoms and departments. According to the chief complaint and demand keyword analysis results, the associated departments related to these symptoms are found in the symptom department association rule base. Through the data in the rule base, the parsed symptoms are automatically matched with the relevant departments to generate the most suitable department recommendation for the user. Structured intent sequence is a series of structured information generated based on keyword analysis and symptom department association rule base, representing the user's intentions, needs, and corresponding recommended departments.

[0047] Intention clustering is to group similar needs or intentions in multiple structured intention sequences. The goal of clustering is to identify patterns or categories with similar user needs through clustering algorithms. For example, a user's chief complaint symptoms include headache, cough, fever, etc. These symptoms can be grouped into similar disease categories, such as cold or nervous system disease. Through clustering, P standardized intention nodes are obtained, each representing a specific demand category or disease type, which can help the system more accurately identify user needs and perform subsequent analysis.

[0048] Through the P standardized intention nodes, multiple structured intention sequences are mapped to a specific guidance path, which represents the conversion process from the user's current needs to the final recommended department. Guidance path mapping involves establishing a relationship between each standardized intention node and medical resources. Each intention node can point to one or more department resource nodes, which are places that can solve the demand.

[0049] The medical resource navigation map is a graph structure, where each node represents a medical resource such as a department, a doctor, or equipment, and the connections between nodes represent their relationships. The medical resource navigation map is constructed based on standardized intention nodes to ensure that each intention can point to the correct medical resource, facilitating smooth medical guidance.

[0050] Further, by using the interaction protocol corresponding to the barrier-free support mode to perform a medical guidance interaction response test on the user, the target medical guidance support mode is screened, and the method comprises:

[0051] If the barrier-free support mode is a single support mode, the barrier-free support mode is used as the target medical guidance support mode. If the barrier-free support mode includes M alternative support modes, after sending multiple minimum interaction instructions to the user using the M alternative support modes, the user's response behavior is collected to obtain M groups of single-mode response features. After performing response confidence analysis on the M groups of single-mode response features, the target medical guidance support mode is selected based on the M modal confidence.

[0052] Single support mode refers to supporting user needs through only one interaction method, such as communicating only through voice or displaying information only through images, which means that when it is identified that the barrier-free support mode is only one, such as a single voice mode or a graphic mode, the mode is directly selected as the target medical guidance support mode. In this case, no further selection or judgment is needed.

[0053] If the barrier-free support mode is not a single mode, but contains M alternative support modes, which refer to multiple interaction methods, such as voice, text, and tactile feedback, in this case, multi-modal interaction testing is performed according to different modes. Specifically, a series of minimum interaction instructions are sent to the user. These instructions are short and targeted interaction prompts, which are used to test the user's response behavior under different modes. For example, the question "Can you hear me speak?" is asked through voice, and a text instruction "Please click the confirmation button" is provided, which aims to understand the user's behavior under different modes through simple and fast interaction testing. Collect the user's response behavior, including the user's reaction to the voice instruction, the click on the text instruction, and the reaction to the tactile feedback. Based on the user's response behavior, collect M sets of single-mode response features for each alternative support mode. Each set of single-mode response features describes the user's response mode under a specific mode, such as whether the user can clearly answer the question, the clarity of the voice, the emotional response of the voice, and whether the user can understand the text information and click the prompt button.

[0054] Response confidence analysis refers to the analysis of the reliability of the user's response under the mode. For example, by analyzing whether the user presses the button, whether the voice is clear, and whether the response is rapid, the response strength and reliability of each mode are determined, and a confidence value is calculated for each mode. This value represents the user's understanding of the interaction mode and the reliability of the response. For example, the confidence of the voice mode is evaluated based on whether the user correctly responds to the voice question, the confidence of the text mode is evaluated based on whether the user clicks the text instruction or feedback, and the confidence of the tactile mode is evaluated based on whether the user has clear tactile feedback. Through the analysis of the confidence of the M modes, the mode with the highest confidence is finally selected as the final target doctor support mode. The goal is to ensure that the selected target doctor support mode is the most suitable for the user's needs and the most effective way of communication, thereby improving user experience and interaction efficiency.

[0055] Further, the method further comprises:

[0056] After performing medical entity extraction on the doctor guidance demand data, a structured query vector is constructed. The structured query vector is used to traverse P standardized intent nodes in the medical resource navigation graph, and the cosine similarity of the P nodes is calculated. The P node cosine similarities are sorted in descending order to extract the initial interaction node with the highest matching degree. The interaction response reply is extracted from the preset reply library of the initial interaction node. The interaction response reply is converted into the modal inquiry response according to the target doctor support mode to reply to the user's initial demand intent.

[0057] The user-provided doctor-guiding demand data includes user symptom description, medical problem, and department demand, etc. The medical entity extraction refers to extracting key information such as symptoms, diseases, and medical needs from the doctor-guiding demand data, and converting the extracted medical entities into a structured query vector. This structured query vector is a mathematical representation that contains all important medical entities and their relationships. The medical entities can be converted into a fixed-dimensional vector representation through methods such as word embedding (e.g. Word2Vec or BERT).

[0058] The structured query vector is used as a search condition to traverse P standardized intent nodes in the medical resource navigation graph, trying to find the most relevant node to the user's demand. In this process, the cosine similarity between each node and the structured query vector is calculated. This is a commonly used method to measure similarity, which measures the angular distance between two vectors. The higher the value, the higher the similarity between the two vectors. Through this calculation, the matching degree of each standardized intent node to the user's demand can be obtained, and P node cosine similarities can be obtained.

[0059] The P node cosine similarities are sorted in descending order. In this way, the node with the highest cosine similarity will be at the front, indicating that these nodes have the strongest matching degree with the user's demand. According to the sorting result, the initial interaction node with the highest cosine similarity is selected. This node represents the best match for the user's demand and will be used as the starting point for further interaction with the user in subsequent steps.

[0060] Each interaction node has a preset reply library that stores standardized replies for the interaction node. These replies are designed by the system to meet user demand and are usually responses based on different user inputs. The preset reply library contains various types of interaction content, such as text, voice, and images. The most relevant interaction response is extracted from the preset reply library of the initial interaction node and used as the interaction response reply, which serves as the basis for subsequent interaction with the user.

[0061] According to the target doctor-guiding support modality, the interaction response reply extracted from the preset reply library of the initial interaction node is converted into the corresponding format. For example, if the target doctor-guiding support modality is voice, the interaction response reply is converted into voice content for playback. If the target doctor-guiding support modality is text and image, the interaction response reply is presented through text and images. After conversion, the modal inquiry response is obtained, which is used to reply to the user's initial demand intent. This reply is the initial feedback provided by the system based on user input and demand, which is used to further guide the user into the doctor-guiding process.

[0062] Furthermore, according to the user's updated feedback demand for the modal inquiry response, multiple rounds of demand intent analysis are performed along the medical resource navigation graph, and a set of department navigation suggestion labels is output. The method comprises:

[0063] After performing medical entity extraction on the update feedback requirement, a structured update vector is constructed; N standardized intent nodes are called from the P standardized intent nodes according to the N interaction sub-nodes of the initial interaction node; an update interaction node is screened according to the cosine similarity between the structured update vector and the N standardized intent nodes; an update response reply is extracted from the preset reply library of the update interaction node; and so on, and according to the feedback requirement of the user on the update response reply, a plurality of rounds of requirement intent analysis are performed along the medical resource navigation graph until the department navigation suggestion label set is output.

[0064] After the initial requirement intent reply of the user is performed according to the modal inquiry response, the user gives further update feedback requirements, which are replies, supplements to the system inquiry, or corrections of the current doctor navigation suggestion by the user, for example, the user feedback “except for headache, I also feel a little fever”, which provides more symptom information for the system. Medical entities are extracted from the update feedback requirement, including new symptoms, disease names, diagnosis and treatment requirements, etc. The structured update vector is a mathematical representation of the medical entities extracted from the update feedback requirement, which converts these entities into structured vectors, so that these information can be used for subsequent analysis and comparison.

[0065] The initial interaction node has N interaction sub-nodes, which represent further refined requirements or symptoms associated with the initial interaction node. By matching with the N interaction sub-nodes, the most relevant nodes are selected from the P standardized intent nodes to obtain N standardized intent nodes. This process further refines the medical intent of the user based on the refined requirements in the user feedback.

[0066] For the N standardized intent nodes, the cosine similarity between them and the structured update vector is calculated. The cosine similarity measures the similarity of two vectors, and the greater the value, the closer their semantics. According to the calculation result of the cosine similarity, the update interaction node with the highest similarity to the structured update vector is selected, which represents the best match of the current user requirement.

[0067] The update interaction node also has a preset reply library containing a series of responses prepared for the node, from which an update response reply is extracted for further communication with the user to ensure continuous and effective interaction according to the feedback provided by the user.

[0068] After receiving the user's feedback again, the above process is repeated, and according to the user's feedback demand in each round, multi-round demand intention analysis is performed, which means that the user's feedback information is repeatedly analyzed, and the user's demand is gradually refined. According to the results of multi-round demand intention analysis, the department navigation suggestion label set is output, which represents the specific department or medical resource recommended for the user. These suggestion labels provide a reference for subsequent manual medical guidance, helping them quickly locate the most suitable department and reducing unnecessary waiting and decision-making time.

[0069] Further, the method further comprises:

[0070] Receiving a resource state push stream of a hospital HIS system; analyzing the resource state push stream to obtain a plurality of department resource occupation characteristics; updating the load state of the plurality of department resource nodes using the plurality of department resource occupation characteristic mapping; based on the M department navigation suggestions in the department navigation suggestion label set, extracting M department resource load states, triggering the available label update of the department navigation suggestion label set based on the load state stability threshold, and outputting an updated department navigation label set.

[0071] The hospital HIS system (hospital information system) is an information management system used by the hospital to manage patient information, resource allocation, department arrangement, etc. It can update the use of various resources in the hospital in real time, such as the occupation state of resources such as departments, doctors, beds, etc. The resource state push stream refers to the real-time state updates about the hospital resources sent by the hospital HIS system, such as which department has spare doctors, which department's beds are occupied, and which medical equipment is in use. By receiving these update information in the form of a push stream in real time, the use of hospital resources can be timely mastered.

[0072] Analyzing the resource state push stream extracts a plurality of department resource occupation characteristics, including the working state of doctors, the use of beds, and the use or idle state of medical equipment.

[0073] The plurality of department resource nodes refers to the representation of each department in the hospital in the medical resource navigation graph. The load state of these nodes is updated using a plurality of department resource occupation characteristics to reflect the current resource usage of the department. For example, if a department's beds are full, the load state of the department is updated to high load; if a department has multiple doctors idle, the load state of the department is updated to low load.

[0074] Based on the M department navigation suggestions in the department navigation suggestion tag set, M department resource load states, i.e. the current load status of each department, such as idle, busy, resource shortage, etc. are extracted. A load state stability threshold is preset, i.e. whether to update the recommended tag is judged according to the change of the department load state, for example, if the load state of a department is always in a high load state, the department will not be recommended to the user temporarily, or its waiting state is recommended; if the load state of a department is eased, such as having a spare bed, the tag of the department will be updated to an available state.

[0075] According to the change of the load state, the department navigation suggestion tag set is updated, and the departments not recommended are removed, or the departments with relatively stable load states and available resources are re-added, for example, the neurology department was not recommended due to full beds, but as the beds are empty, it will be re-added to the recommended list. Finally, the department navigation update tag set is output, which contains the departments most suitable for recommendation at present, providing a reference for artificial doctor navigation.

[0076] Further, the time sequence image array and the time sequence audio information are used for doctor navigation support state detection, and a barrier-free support mode is output, and the method comprises:

[0077] Human body pose key point detection is performed on the time sequence image array to obtain a plurality of limb action sequences; a light flow vector amplitude calculation of a continuous frame hand exploration action is performed on the plurality of limb action sequences to output a limb exploration feature vector; auxiliary equipment feature detection is performed on the time sequence image array to output an auxiliary equipment feature vector; acoustic feature detection is performed on the time sequence audio information to output an audio feature vector; the limb exploration feature vector, the auxiliary equipment feature vector and the audio feature vector are fused and input into a multi-task classifier, and the multi-task classifier is used for judgment to output the barrier-free support mode.

[0078] The time sequence image array is a continuous image sequence obtained by an image acquisition device, and the images are arranged in time sequence and represent the dynamic behavior of the user in a certain period of time. Human body pose key point detection refers to identifying the key points of the human body in the image and determining their positions through computer vision technology. These key points include various parts of the body, such as the head, shoulders, elbows, wrists, knees and ankles, etc. By detecting these key points, the motion trajectory and posture change of the human body can be analyzed, for example, the moving path of the user's hand is detected, or it is analyzed whether the user has made a specific action, such as raising the hand, waving the hand, walking, etc. After the human body pose key point detection is performed on the time sequence image array, a plurality of limb action sequences are extracted therefrom, which are composed of time sequence data of key points and describe the limb actions of the user in the time dimension.

[0079] Optical flow refers to the speed of relative change over time of the surface of objects in an image, commonly used to describe the movement of pixels in an image. For limb movements, optical flow can help capture the displacement and speed changes of the hand or other limb parts in the image. Hand exploration action refers to the movement of the user's hand during the execution of the action, such as waving, pointing, grabbing, etc. By analyzing consecutive frames, the movement of the hand between different time points is identified. Further, the optical flow vector amplitude of the hand exploration action is calculated, i.e. the displacement and speed of the hand are calculated through the pixel difference between consecutive two frames. The optical flow vector amplitude refers to the size of the optical flow vector, indicating the speed of the hand movement within a time interval. The larger the optical flow vector amplitude, the faster the hand movement. By calculating the optical flow vector amplitude, the intensity and direction of the hand exploration action can be captured, for example, if the user's hand moves quickly within a certain period of time, the amplitude of the optical flow will be larger, and vice versa. Through the calculation of the optical flow vector amplitude, the limb exploration feature vector is obtained, which describes the dynamic characteristics of the user's hand exploration action, including the direction of hand movement, the speed of hand movement, etc.

[0080] In the time sequence image array, the target of auxiliary device feature detection is to identify whether there is an auxiliary device used by the user in the image, such as a walking stick, a wheelchair, a hearing aid, etc. These auxiliary devices are usually related to the specific needs or conditions of the user, such as visual impairment, mobility impairment or hearing impairment. Through image recognition technology, such as convolutional neural network, each frame of image is analyzed to check whether the visual features of these auxiliary devices appear in the image. For example, a walking stick is usually a long, upright device, and its position, shape and usage can be identified when held by the user. After detection, an auxiliary device feature vector is output, which contains key information about the auxiliary device, helping to understand the user's demand type.

[0081] Time sequence audio information is the sound information of the user collected by audio collection devices such as microphones. Acoustic feature detection is to analyze the audio signal and extract features that can describe the sound. Through acoustic feature detection, an audio feature vector is obtained, which contains key information of the audio signal, such as speech clarity, volume, tone, etc., which can help determine the user's state or needs.

[0082] The limb exploration feature vector, auxiliary device feature vector and audio feature vector are multiple feature vectors extracted from image and audio data respectively. These vectors describe the user's limb movement, the auxiliary device used and the sound information respectively. Combining these three different types of feature vectors into a comprehensive vector can comprehensively reflect the user's needs and state.

[0083] The multi-task classifier is a model capable of processing multiple related tasks, which analyzes based on the fused feature vector, determines the user's demand type, and outputs the most suitable barrier-free support mode, for example, voice interaction is suitable for users with clear language expression, and tactile feedback or other barrier-free support is suitable for users with physical or hearing disabilities.

[0084] Further, the acoustic feature detection is performed on the time-series audio information to output an audio feature vector, and the method comprises:

[0085] Endpoint detection is performed on the time-series audio information to segment the valid audio segment; the signal-to-noise ratio feature of the valid audio segment is identified by combining the mel-frequency cepstral coefficient and the spectral entropy analysis; the high-frequency energy loss feature is obtained by performing high-frequency segment energy detection on the valid audio segment; after the time-series translation text is obtained by performing text translation on the valid audio segment, the confidence difference feature is obtained by solving the confidence difference between the time-series translation text and the valid audio segment; and the signal-to-noise ratio feature, the high-frequency energy loss feature, and the confidence difference feature are fused to output the audio feature vector.

[0086] Endpoint detection refers to identifying the valid part in the time-series audio information, i.e., segmenting the valid audio segment from the audio, and only focusing on the meaningful part in the audio signal, such as the user's voice, in the acoustic feature detection process, and not focusing on the background noise or silent part. Endpoint detection can automatically remove these irrelevant parts, thereby retaining only useful audio information. In this process, the start and end points of the audio are detected by the features of the audio signal, such as volume change, spectral change, etc. For example, a minimum energy threshold is set to detect the start and end points of the audio signal. When the audio signal exceeds the threshold, it is a valid audio segment, and when it is below the threshold, it is a silent or noisy part. According to these endpoints, the valid audio segment is segmented, which contains the user's voice information or other related sound signals, and does not contain environmental noise or blank parts.

[0087] Mel-frequency cepstral coefficient is a commonly used feature in speech signal processing, which can reflect the spectral information of sound signals. Mel-frequency cepstral coefficient analyzes audio signals by simulating the hearing characteristics of the human ear and is commonly used in speech recognition, emotion analysis and other applications. Each valid audio segment is converted into a mel-frequency cepstral coefficient feature to reflect the spectral structure of the sound. The mel-frequency cepstral coefficient feature can capture key information such as pitch and speech intelligibility of the audio signal.

[0088] Spectral entropy is an indicator to measure the complexity of audio signals. It determines the entropy value of the signal by analyzing the spectral distribution of the audio signal. The higher the entropy value, the more complex or noisy the signal is. The lower the entropy value, the simpler the signal and the less noise.

[0089] The signal-to-noise ratio feature of the effective audio segment is identified by combining the mel-frequency cepstral coefficient and the spectrum entropy analysis. The signal-to-noise ratio represents the ratio of useful signals to noise. A higher signal-to-noise ratio means that the audio signal is clearer and has less noise. In this way, the signal-to-noise ratio feature of the effective audio segment can be identified to help judge the quality and intelligibility of the audio signal.

[0090] High-frequency energy detection refers to analyzing the frequency band above 8000 Hz in the audio signal. The high-frequency part usually contains important audio information in the speech signal, such as clarity and detailed parts of the speech. The high-frequency part of the audio signal is extracted by a filter, and the energy of these frequency bands is calculated to obtain the high-frequency energy loss feature. The high-frequency energy loss feature has a high-frequency energy loss identifier, which is a binary identifier indicating whether the high-frequency part of the audio signal has energy loss. If the energy of the high-frequency part is below a certain threshold, it is marked as 1, otherwise it is marked as 0.

[0091] Text translation is the process of converting speech information in the effective audio segment into text information. Speech recognition technology, such as automatic speech recognition, is used to process the effective audio segment and extract corresponding text information to obtain time-ordered translation text. In the text translation process, a corresponding confidence is generated for the time-ordered translation text, indicating the credibility of the translation result. Confidence difference refers to the matching degree between the actual speech content of the audio signal and the translated time-ordered translation text. If the text translation is accurate, the confidence difference is small; if the text translation has errors, the confidence difference is large. For example, the difference between the translated text and the actual speech content is calculated to obtain the word error rate. The lower the word error rate, the smaller the confidence difference, indicating that the translated text is more accurate. By comparing the time-ordered translation text and the speech content of the effective audio segment, the confidence difference between them is calculated to obtain the confidence difference feature.

[0092] The signal-to-noise ratio feature, high-frequency energy loss feature, and confidence difference feature extracted earlier are fused to obtain a comprehensive audio feature vector. This audio feature vector contains multi-dimensional information about audio quality and speech recognition accuracy, and can comprehensively describe the quality of the audio signal.

[0093] In summary, the multi-modal interaction-based barrier-free intelligent doctor guiding method provided by the embodiments of the present application has the following technical effects:

[0094] The time sequence state data of the user is collected in real time by the dual-mode state acquisition unit, and the doctor guiding support state detection is automatically performed based on the data, so that the demand type of the user can be intelligently judged, and the most suitable doctor guiding support mode can be selected according to the physical condition and interaction preference of the user, thereby improving the individualization and intelligent level of the medical experience; by calling the corresponding target mode doctor guiding interaction container according to the target doctor guiding support mode, the user can be provided with an interactive mode suitable for his / her needs, and personalized interactive modes can be provided for different user groups, thereby significantly improving the adaptability of the intelligent doctor guiding system to special groups; by traversing the medical resource navigation graph based on the doctor guiding demand data input by the user, the initial interaction node is located, and by using the intelligent medical resource navigation graph, the most suitable department recommendation can be quickly responded and provided, thereby reducing the navigation time and unnecessary consultation of the user in the hospital and improving the optimization scheduling and use efficiency of the hospital resources; after extracting the interaction response reply of the initial interaction node, the interaction response reply is converted into a modal inquiry response meeting the needs of the user according to the target doctor guiding support mode, and the intelligent adaptation enhances the interactivity and user experience of the system, especially in the user groups with obstacles or special needs, the self-help ability of the users can be maximized and the waiting and information acquisition time can be reduced; according to the updated feedback needs of the user, multi-round demand intention analysis is performed along the medical resource navigation graph, the department navigation suggestion label is accurately optimized, and by continuously iterating the analysis, the recommended department can be dynamically adjusted, especially for users with changing needs or complex symptoms, the recommended path can be continuously optimized, and the possibility of recommendation errors or resource waste can be effectively reduced; by packaging the demand interaction record and the department navigation suggestion label set, they are sent to the target auxiliary doctor, and the target auxiliary doctor makes a decision and performs guiding service according to the data, the mechanism provides accurate data support to help the target auxiliary doctor make a quick decision and optimize the department scheduling and user guidance, especially in the case of resource shortage, the utilization efficiency of medical resources can be effectively improved, and the waiting time of the user can be reduced.

[0095] In the embodiment two, based on the same inventive concept as the multi-modal interaction based barrier-free intelligent doctor guiding method in the foregoing embodiments, as shown in the following table, the application embodiment provides a multi-modal interaction based barrier-free intelligent doctor guiding platform, which comprises: Figure 2 The state detection module 10 is configured to perform doctor guiding support state detection according to the time sequence state data of the user acquired by the dual-mode state acquisition unit, and output a target doctor guiding support mode.

[0096] The container calling module 20 is configured to call a target mode doctor guiding interaction container in the multi-modal doctor guiding interaction platform according to the target doctor guiding support mode.

[0097]

[0098] ​The interactive node positioning module 30 is configured to use the user inputted doctor guidance demand data to traverse the medical resource navigation graph to position an initial interactive node.

[0099] The intention reply module 40 is configured to convert the interactive reply into a modal inquiry response according to the target doctor guidance support mode after extracting the interactive reply of the initial interactive node to reply to the user's initial demand intention.

[0100] The intention analysis module 50 is configured to perform multi-round demand intention analysis along the medical resource navigation graph according to the user's updated feedback demand for the modal inquiry response, and output a department navigation suggestion label set.

[0101] The alarm sending module 60 is configured to package the demand interactive record and the department navigation suggestion label set into a doctor guidance demand alarm and send the alarm to the target auxiliary doctor guidance according to the target doctor guidance support mode.

[0102] Further, the state detection module 10 is configured to perform the following operation steps:

[0103] When the user's stay in the doctor guidance island platform monitoring area meets a preset threshold, the double-mode state acquisition unit is activated to acquire time sequence state data of the user; the time sequence state data is analyzed to obtain a time sequence image array and time sequence audio information; the time sequence image array and the time sequence audio information are used for doctor guidance support state detection to output an accessible support mode; the target doctor guidance support mode is screened by performing a doctor guidance interactive response test on the user through an interactive protocol corresponding to the accessible support mode.

[0104] Further, the interactive node positioning module 30 is configured to perform the following operation steps:

[0105] The local call multi-group single-mode doctor guidance dialogue sequence is called, wherein the single-mode doctor guidance dialogue sequence is desensitized; the multi-group single-mode doctor guidance dialogue is cross-modal semantic aligned to generate a plurality of doctor coding time sequence feature vectors; after the plurality of doctor coding time sequence feature vectors are analyzed for chief complaint demand keywords, a plurality of structured intention sequences are generated in combination with a symptom department association rule base; the plurality of structured intention sequences are intention clustered to obtain P standardized intention nodes; the plurality of structured intention sequences are mapped for doctor diagnosis paths according to the P standardized intention nodes to construct the medical resource navigation graph, wherein the last layer node of the medical resource navigation graph is a plurality of department resource nodes.

[0106] Further, the state detection module 10 is configured to perform the following operation steps:

[0107] If the barrier-free support mode is a single support mode, the barrier-free support mode is taken as the target doctor-guiding support mode; if the barrier-free support mode includes M alternative support modes, after sending multi-element minimum interaction instructions to the user by using the M alternative support modes, user response behavior collection is performed to obtain M groups of single-mode response features; after performing response confidence analysis on the M groups of single-mode response features, the target doctor-guiding support mode is obtained by screening according to M mode confidences.

[0108] Further, the intention reply module 40 is configured to perform the following operation steps:

[0109] After performing medical entity extraction on the doctor-guiding demand data, a structured query vector is constructed; the structured query vector is used to traverse P standardized intention nodes in the medical resource navigation graph, and P node cosine similarities are calculated; the P node cosine similarities are arranged in descending order to extract the initial interaction node with the highest matching degree; the interaction reply is extracted from the preset reply library of the initial interaction node; and the interaction reply is converted into the modal interrogation response according to the target doctor-guiding support mode to reply to the initial demand intention of the user.

[0110] Further, the intention analysis module 50 is configured to perform the following operation steps:

[0111] After performing medical entity extraction on the updated feedback demand, a structured update vector is constructed; N standardized intention nodes are called from the P standardized intention nodes according to N interaction sub-nodes of the initial interaction node; an updated interaction node is screened and positioned according to the structured update vector and the cosine similarity of the N standardized intention nodes; an updated reply is extracted from the preset reply library of the updated interaction node; and in this way, multi-round demand intention analysis is performed along the medical resource navigation graph according to the feedback demand of the user on the updated reply, until the department navigation suggestion label set is output.

[0112] Further, the intention analysis module 50 is configured to perform the following operation steps:

[0113] A resource state push stream of a hospital HIS system is received; a plurality of department resource occupation features are obtained by analyzing the resource state push stream; load state updating of the plurality of department resource nodes is performed by using the plurality of department resource occupation feature mapping; after extracting M department resource load states according to M department navigation suggestions in the department navigation suggestion label set, available label updating of the department navigation suggestion label set is triggered based on a load state stability threshold, and a department navigation update label set is output.

[0114] Further, the state detection module 10 is configured to perform the following operation steps:

[0115] The time sequence image array is subjected to human pose key point detection to obtain a plurality of limb motion sequences; the plurality of limb motion sequences are subjected to light flow vector amplitude calculation of a continuous frame-to-frame hand exploration motion, and a limb exploration feature vector is output; the time sequence image array is subjected to auxiliary equipment feature detection, and an auxiliary equipment feature vector is output; the time sequence audio information is subjected to acoustic feature detection, and an audio feature vector is output; the limb exploration feature vector, the auxiliary equipment feature vector and the audio feature vector are fused and input into a multi-task classifier, and the multi-task classifier is used to determine and output the barrier-free support mode.

[0116] Further, the state detection module 10 is configured to perform the following operation steps:

[0117] The time sequence audio information is subjected to endpoint detection to segment an effective audio segment; a signal-to-noise ratio feature of the effective audio segment is identified by combining a mel-frequency cepstrum coefficient and a spectrum entropy analysis; the effective audio segment is subjected to high-frequency band energy detection to obtain a high-frequency energy loss feature; after a time sequence translation text is obtained by subjecting the effective audio segment to text translation, the time sequence translation text and the effective audio segment are subjected to confidence difference solving to obtain a confidence difference feature; and the signal-to-noise ratio feature, the high-frequency energy loss feature and the confidence difference feature are fused to output the audio feature vector.

[0118] Through the foregoing detailed description of the barrier-free intelligent doctor guiding method based on multi-modal interaction, those skilled in the art can clearly understand the barrier-free intelligent doctor guiding platform based on multi-modal interaction in the embodiments. Since the platform corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant part can be referred to the method part description.

[0119] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A barrier-free intelligent doctor guiding method based on multi-modal interaction, characterized in that, The method comprises: According to the bimodal state acquisition unit, the user's time sequence state data is acquired to detect the doctor support state, and a target doctor support mode is output; According to the target doctor support mode, a target mode doctor interaction container is called on a multi-modal doctor interaction platform; The target mode doctor interaction container uses the doctor demand data input by the user to traverse the medical resource navigation graph to locate an initial interaction node; After extracting the interaction response reply of the initial interaction node, the target doctor support mode is used to convert the interaction response reply into a modal inquiry response to reply to the user's initial demand intention; According to the updated feedback demand of the user to the modal inquiry response, a plurality of rounds of demand intention analysis are performed along the medical resource navigation graph, and a department navigation suggestion label set is output; After matching the target auxiliary doctor according to the target doctor support mode, the demand interaction record and the department navigation suggestion label set are packaged as a doctor demand alarm and sent to the target auxiliary doctor; According to the bimodal state acquisition unit, the user's time sequence state data is acquired to detect the doctor support state, and a target doctor support mode is output, the method comprising: When the user stays in the doctor island platform monitoring area for a duration that meets a preset threshold, the bimodal state acquisition unit is activated to acquire the user's time sequence state data; The time sequence state data is analyzed to obtain a time sequence image array and time sequence audio information; The time sequence image array and time sequence audio information are used for doctor support state detection, and a barrier-free support mode is output; The target doctor support mode is screened by performing a doctor interaction response test on the user using the interaction protocol corresponding to the barrier-free support mode; The target doctor support mode is screened by performing a doctor interaction response test on the user using the interaction protocol corresponding to the barrier-free support mode, the method comprising: If the barrier-free support mode is a single support mode, the barrier-free support mode is used as the target doctor support mode; If the barrier-free support mode includes M alternative support modes, after sending multiple minimum interaction instructions to the user using the M alternative support modes, the user's response behavior is collected to obtain M groups of single-mode response features; After response confidence analysis is performed on the M groups of single-mode response features, the target doctor support mode is screened according to M modal confidence.

2. The multi-modal interaction based barrier-free smart doctor guiding method as claimed in claim 1, wherein, The method further comprises: locally calling a plurality of single-mode doctor dialogue sequences, wherein the single-mode doctor dialogue sequences are desensitized; performing cross-modal semantic alignment on the plurality of single-mode doctor dialogues to generate a plurality of doctor coding time sequence feature vectors; After analyzing the key words of the plurality of structured intention sequences, a plurality of structured intention sequences are generated in combination with the symptom department association rule base; The plurality of structured intention sequences are clustered to obtain P standardized intention nodes; According to the P standardized intention nodes, the plurality of structured intention sequences are mapped to a doctor diagnosis path to construct the medical resource navigation graph, wherein the last layer node of the medical resource navigation graph is a plurality of department resource nodes.

3. The multi-modal interaction based barrier-free smart doctor guiding method as claimed in claim 2, wherein, The method further comprises: After medical entity extraction is performed on the doctor guidance demand data, a structured query vector is constructed; The structured query vector is used to traverse P standardized intent nodes in the medical resource navigation graph, and P node cosine similarities are calculated; The P node cosine similarities are arranged in descending order to extract the initial interaction node with the highest matching degree; The interaction response reply is extracted from the preset reply library of the initial interaction node; According to the target doctor guidance support mode, the interaction response reply is converted into the mode inquiry response for the initial demand intent reply of the user.

4. The multi-modal interaction based barrier-free smart doctor guiding method as claimed in claim 2, wherein, According to the update feedback demand of the user on the mode inquiry response, multi-round demand intent analysis is performed along the medical resource navigation graph, and a department navigation suggestion label set is output, the method comprising: After medical entity extraction is performed on the update feedback demand, a structured update vector is constructed; According to the N interaction sub-nodes of the initial interaction node, N standardized intent nodes are called from the P standardized intent nodes; According to the cosine similarity between the structured update vector and the N standardized intent nodes, the update interaction node is located and screened; The update response reply is extracted from the preset reply library of the update interaction node; By analogy, according to the feedback demand of the user on the update response reply, multi-round demand intent analysis is performed along the medical resource navigation graph, until the department navigation suggestion label set is output.

5. The multi-modal interaction based barrier-free smart doctor guiding method as claimed in claim 2, wherein, The method further comprises: Receiving a resource state push stream of a hospital HIS system; Obtaining a plurality of department resource occupation characteristics by analyzing the resource state push stream; Using the plurality of department resource occupation characteristics mapping to update the load state of the plurality of department resource nodes; According to the M department navigation suggestions in the department navigation suggestion label set, the M department resource load states are extracted, and based on the load state stability threshold, the available label update of the department navigation suggestion label set is triggered, and a department navigation update label set is output.

6. The multi-modal interaction based barrier-free smart doctor guiding method as claimed in claim 1, wherein, Using the time sequence image array and the time sequence audio information to detect the doctor guidance support state, and outputting the barrier-free support mode, the method comprising: Performing human body pose key point detection on the time sequence image array to obtain a plurality of limb action sequences; Performing optical flow vector amplitude calculation of continuous frame hand exploration actions on the plurality of limb action sequences to output a limb exploration feature vector; Performing auxiliary equipment feature detection on the time sequence image array to output an auxiliary equipment feature vector; Performing acoustic feature detection on the time sequence audio information to output an audio feature vector; Fusing the limb exploration feature vector, the auxiliary equipment feature vector and the audio feature vector into a multi-task classifier, and outputting the barrier-free support mode through the multi-task classifier.

7. The multi-modal interaction based barrier-free smart doctor guiding method as claimed in claim 6, wherein, Performing acoustic feature detection on the time sequence audio information to output an audio feature vector, the method comprising: Performing endpoint detection on the time sequence audio information to segment valid audio segments; Combining mel-frequency cepstrum coefficient and spectrum entropy analysis to identify the signal-to-noise ratio feature of the valid audio segment; Performing high-frequency band energy detection on the valid audio segment to obtain a high-frequency energy loss feature; The effective audio segment is text translated to obtain time sequence translated text, and the time sequence translated text and the effective audio segment are solved to obtain a confidence difference feature; The signal-to-noise ratio feature, the high-frequency energy loss feature, and the confidence difference feature are fused to output the audio feature vector.

8. Barrier-free intelligent doctor guiding platform based on multi-modal interaction, characterized in that, The platform comprises: A state detection module configured to acquire user time sequence state data from a bimodal state acquisition unit to perform medical guide support state detection and output a target medical guide support mode; A container calling module configured to call a target mode medical guide interaction container in a multimodal medical guide interaction platform according to the target medical guide support mode; An interaction node positioning module configured to use user input medical guide demand data to traverse a medical resource navigation graph to position an initial interaction node in the target mode medical guide interaction container; An intention reply module configured to convert an interaction response reply of the initial interaction node into a mode inquiry response according to the target medical guide support mode to reply to a user initial demand intention after the interaction response reply is extracted; An intention analysis module configured to perform multi-round demand intention analysis along the medical resource navigation graph according to user updated feedback demand for the mode inquiry response to output a department navigation suggestion label set; An alarm sending module configured to package demand interaction records and the department navigation suggestion label set into a medical guide demand alarm and send the medical guide demand alarm to a target auxiliary medical guide according to the target medical guide support mode after the target auxiliary medical guide is matched.

Images