Multi-modal fusion in-hospital intelligent robot doctor-seeing guiding system and multi-modal fusion in-hospital intelligent robot doctor-seeing guiding method

Abstract

The invention discloses a multi-modal fusion hospital intelligent robot doctor-seeing guiding system and method, relates to the technical field of intelligent medical treatment, and solves the problems that in the prior art, dynamic factors are not considered during path planning, and perception and self-adaptive adjustment on the emotion of a patient are not available, so that the accuracy and efficiency of a doctor-seeing guiding system are low, and the service life of the doctor-seeing guiding system is short. And the satisfaction degree of the doctor seeing personnel is relatively low. A suggested doctor-seeing department is generated according to a detection image and voice data, and an optimal guiding path is generated accordingly; an interaction strategy is generated according to the voice data and the body parameters, an alarm signal is generated, a doctor-seeing department is rapidly positioned by combining doctor-seeing person description and historical doctor-seeing medical records, an optimal path is constructed according to the current position and the position of the doctor-seeing department, time consumed by the doctor-seeing person for asking the way is shortened, and the doctor-seeing person can arrive at the doctor-seeing department as soon as possible; meanwhile, the robot interaction strategy is adaptively adjusted according to the state of the patient, the negative emotion of the patient is effectively relieved, and the user satisfaction is improved.

Description

Technical Field

[0001] This application belongs to the field of intelligent medical technology, and specifically relates to a multi-modal fusion intelligent robot medical treatment guidance system and method in a hospital. Background Technique

[0002] A guiding robot is an intelligent device with functions such as autonomous navigation, voice interaction, and intelligent recognition. It is widely used in public places such as hospitals, shopping malls, airports, and hotels to provide users with convenient navigation, information query, and interaction services. In a hospital environment, guiding robots are particularly important as they can significantly improve the medical experience of patients and optimize the service process of the hospital.

[0003] Currently, there are already intelligent robot medical treatment guidance systems in some hospitals, but there are still some limitations. For example, the path planning is static, relying on a fixed map, unable to respond to dynamic factors such as the density of the flow of people in real time. And when interacting with medical treatment personnel, it can only provide basic questions and answers, lacking the perception and adaptive adjustment of patients' emotions and physiological states, resulting in low accuracy and efficiency of the medical treatment guidance system and low satisfaction of medical treatment personnel. Therefore, the intelligent robot medical treatment guidance system still needs further improvement. Summary of the Invention

[0004] This application aims to at least solve one of the technical problems existing in the prior art. For this purpose, this application proposes a multi-modal fusion intelligent robot medical treatment guidance system and method in a hospital, which is used to solve the technical problems that the prior art lacks consideration of dynamic factors during path planning and lacks the perception and adaptive adjustment of patients' emotions, resulting in low accuracy and efficiency of the medical treatment guidance system and low satisfaction of medical treatment personnel.

[0005] To achieve the above object, the first aspect of this application provides a multi-modal fusion intelligent robot medical treatment guidance system in a hospital, including: a data acquisition module, a data analysis module, an early warning module, and a database;

[0006] The data acquisition module: obtains medical treatment personnel data, hospital comprehensive data, and robot data through data acquisition devices; the medical treatment personnel data includes personnel ID, voice data, physical parameters, and detection images; the hospital comprehensive data includes several department IDs, department locations, and comprehensive monitoring videos;

[0007] The data analysis module: generates a recommended department for medical treatment based on the detection images and voice data; generates an optimal guiding path based on the recommended department for medical treatment and the comprehensive monitoring video; generates an interaction strategy and an alarm signal based on the voice data and physical parameters;

[0008] The early warning module: makes a prompt according to the alarm signal and contacts the management personnel;

[0009] The database is used to store the data of each module and the historical data required for training the model.

[0010] Through the above steps, this application organically integrates the condition descriptions of the patients seeking medical treatment with the information of their previous medical records, so as to quickly determine the department they should go to; on this basis, the system will plan an optimal path according to the current location of the patient seeking medical treatment and the location of the department, so as to reduce the time consumed in the process of finding the way and ensure that they can arrive at the target department as soon as possible; in addition, the system will also dynamically adjust the interaction strategy of the robot according to the real-time status of the patient seeking medical treatment, so as to effectively relieve the negative emotions such as anxiety of the patient seeking medical treatment and further improve the user satisfaction.

[0011] Further, generating the recommended department for medical treatment according to the detected image and voice data includes:

[0012] Obtain the detected image and voice data;

[0013] Input the detected image into the medical record recognition table to obtain the historical medical record labels and their corresponding medical record contents; the medical record recognition table is constructed by experts according to the historical medical record contents of the hospital's previous patients seeking medical treatment;

[0014] Integrate the voice data, the historical medical record labels and their corresponding medical record contents into recommended data;

[0015] Input the recommended data into the medical treatment recommendation model to obtain the recommended department for medical treatment; the medical treatment recommendation model is constructed by a large model.

[0016] Further, constructing the medical treatment recommendation model by a large model includes:

[0017] Obtain a number of historical recommended data and their corresponding historical recommended departments for medical treatment;

[0018] Divide a number of historical recommended data and their corresponding historical recommended departments for medical treatment into training data, validation data and test data; perform data preprocessing on the training data, validation data and test data to obtain a training set, a validation set and a test set;

[0019] Select a large model as the basic model;

[0020] Train the basic model with the training set, and adjust the learning rate and hyperparameters on the validation set to obtain a pre-trained model;

[0021] Verify the pre-trained model on the test set, and finally obtain a medical treatment recommendation model that inputs recommended data and outputs the recommended department for medical treatment.

[0022] Further, generating an optimal guiding path based on the recommended department for consultation and the comprehensive monitoring video includes:

[0023] Obtaining the personnel location corresponding to the person seeking medical treatment, the comprehensive monitoring video, and the department location corresponding to the recommended department for consultation in real time;

[0024] Obtaining several optional paths in the path table based on the personnel location and the department location; the path table is set by experts according to historical optional paths;

[0025] Generating several congestion indices based on the comprehensive monitoring video;

[0026] Obtaining the congestion weights and distance weights corresponding to several optional paths after performing max-min normalization on several congestion indices and the path distances corresponding to several optional paths;

[0027] Selecting the optional path corresponding to the minimum value of the sum of the congestion weight and the distance weight as the optimal path.

[0028] Further, generating several congestion indices based on the comprehensive monitoring video includes:

[0029] Obtaining several comprehensive monitoring videos; the comprehensive monitoring video refers to the monitoring video corresponding to the optional path;

[0030] Extracting the crowd density RM, the movement attenuation coefficient YJ, the stagnation time ratio TT, and the monitoring time from the comprehensive monitoring video;

[0031] Through the formula Calculating the congestion index YZ i ; where, β1, β2, and β3 are adjustment coefficients, and β1, β2, and β3 are all greater than 0; the adjustment coefficients are adaptively adjusted according to the crowd density, the monitoring time, and the stagnation time ratio.

[0032] Further, the adjustment coefficients being adaptively adjusted according to the crowd density, the monitoring time, and the stagnation time ratio includes:

[0033] Obtaining the crowd density, the monitoring time, and the stagnation time ratio corresponding to several optional paths;

[0034] Integrating the crowd density, the monitoring time, and the stagnation time ratio corresponding to several optional paths into adjustment coefficient data;

[0035] Inputting the adjustment coefficient data into the adjustment coefficient prediction model to obtain several adjustment coefficients corresponding to several optional paths;

[0036] The adjustment coefficient prediction model is constructed through an artificial intelligence model, including:

[0037] Obtain a number of historical adjustment coefficient data and their corresponding historical adjustment coefficients;

[0038] Divide a number of historical adjustment coefficient data and their corresponding historical adjustment coefficients into training data, validation data, and test data; perform data preprocessing on the training data, validation data, and test data to obtain a training set, a validation set, and a test set;

[0039] Select an artificial intelligence model as the basic model;

[0040] Train the basic model with the training set, and adjust the learning rate and hyperparameters on the validation set to obtain a pre-trained model;

[0041] Verify the pre-trained model on the test set, and finally obtain an adjustment coefficient prediction model with input adjustment coefficient data and output of a number of adjustment coefficients.

[0042] This application obtains each parameter in a number of optional paths in real time, which is used to quantify the congestion degree corresponding to the optional path, and adaptively obtains a number of adjustment coefficients for calculating the congestion index in different optional paths according to different parameter values in each optional path, so that the adjustment coefficient can change with time and scenarios, avoiding the mechanicalness of static formulas, improving the accuracy of the congestion index of the optional path, and providing strong data support for path guidance.

[0043] Further, the generating an interaction strategy and generating an alarm signal according to the voice data and body parameters includes:

[0044] Obtain voice data and body parameters; the voice data includes intonation, and the body parameters include age and heart rate;

[0045] Through the formula Calculate the tension coefficient of the patient; where DX represents the current heart rate, JX represents the resting heart rate; YDB represents the standard deviation of the pitch, DP represents the jitter frequency, BYD represents the standard pitch value, and BDP represents the standard jitter frequency; the resting heart rate and the standard deviation of the pitch are calculated according to age and intonation;

[0046] When the tension coefficient is within the normal range, do nothing;

[0047] When the tension coefficient is within the tense range, generate a patient tension warning signal and generate an interaction strategy according to the tension coefficient.

[0048] Further, the calculating the resting heart rate and the standard deviation of the pitch according to age and intonation includes:

[0049] Obtain age and intonation; the intonation includes pitch values at consecutive N time points; where N is an integer and N>0;

[0050] The resting heart rate is equal to the average of several resting heart rates within the age range to which the person belongs;

[0051] The tone average value YDP is obtained by taking the average of several tone values;

[0052] Through the formula Calculate the tone standard deviation YDB; where, YD n Represents the tone value at the nth time point between the current times.

[0053] Furthermore, the generating of the interaction strategy according to the tension coefficient includes:

[0054] Obtain the tension coefficient and the robot data; the robot data includes the interaction frequency and the interaction mode; the interaction mode includes the normal mode and the soothing mode;

[0055] When the tension coefficient is within the tension range, set the interaction mode to the soothing mode;

[0056] Through the formula Calculate the interaction frequency JP; where, JP min Represents the interaction frequency in the normal interaction mode; ΔJP represents the maximum growth rate of the interaction frequency; k represents the tension response coefficient, k>0; DJX represents the unit tension coefficient.

[0057] Another aspect of the present invention provides a method for guiding the visit of an intelligent robot in a hospital with multi-modal fusion, including:

[0058] S0: Obtain the visitant data, the hospital comprehensive data and the robot data; the visitant data includes the person ID, the voice data, the physical parameters and the detection images; the hospital comprehensive data includes several department IDs, the department locations and the comprehensive monitoring videos;

[0059] S1: Generate the recommended department for the visit according to the detection images and the voice data;

[0060] S2: Generate the optimal guiding path according to the recommended department for the visit and the comprehensive monitoring videos;

[0061] S3: Generate the interaction strategy according to the voice data and the physical parameters and generate an alarm signal;

[0062] S4: Make a prompt according to the alarm signal and contact the management personnel.

[0063] Compared with the prior art, the beneficial effects of the present application are:

[0064] 1. This application generates a recommended department for medical treatment based on the detected image and voice data; generates an optimal guiding path based on the recommended department for medical treatment and the comprehensive monitoring video; generates an interaction strategy and an alarm signal based on the voice data and physical parameters, combines the description of the patient and the historical medical records, quickly locates the department for medical treatment, constructs the optimal path based on the current location and the location of the department for medical treatment, reduces the time consumed by the patient asking for directions, and reaches the department for medical treatment as soon as possible; at the same time, adaptively adjusts the robot interaction strategy according to the patient's state, effectively relieves the negative emotions of the patient, and improves user satisfaction.

[0065] 2. This application generates several optional paths based on the location of the patient and the location of the recommended department for medical treatment, comprehensively considers the path distance and path congestion degree among the several optional paths, and selects the optimal optional path as the guiding path, so that the patient can quickly reach the department for medical treatment, improving the guiding accuracy and guiding efficiency of the medical treatment guiding system.

[0066] 3. This application, during the guiding process, real-time monitors the physical state and psychological state of the patient. When it detects that the patient has a tense psychology, the guiding robot can adaptively provide comfort services and change the interaction frequency, enabling the patient to reduce the tense emotion, providing a more user-friendly service, and improving the satisfaction of the patient. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0068] Figure 1 It is a schematic diagram of the principle of an intelligent robot medical treatment guiding system with multi-modal fusion in the hospital according to the present application;

[0069] Figure 2 It is a flowchart of a method for an intelligent robot medical treatment guiding system with multi-modal fusion in the hospital according to the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0070] The following will clearly and completely describe the technical solutions of the present application in combination with the embodiments. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0071] Please refer to Figure 1, in the first aspect embodiment of the present application, an intelligent robot medical guidance system for in-hospital multi-modal fusion is provided, including: a data acquisition module, a data analysis module, an early warning module, and a database;

[0072] Data acquisition module: Obtain medical personnel data, hospital comprehensive data, and robot data through data acquisition devices; The medical personnel data includes personnel ID, voice data, physical parameters, and detection images. The detection image refers to the facial acquisition image of the medical personnel; The hospital comprehensive data includes several department IDs, department locations, and comprehensive monitoring videos; The data acquisition devices include several sensors, etc.

[0073] Data analysis module: Generate a recommended department for medical treatment according to the detection image and voice data. The recommended department for medical treatment refers to the department recommended for the medical personnel to go for examination; Generate the optimal guidance path according to the recommended department for medical treatment and the comprehensive monitoring video. The optimal guidance path refers to the optimal path for the medical personnel to go to the recommended department for medical treatment; Generate an interaction strategy and an alarm signal according to the voice data and physical parameters. The interaction strategy includes interaction mode, interaction frequency, etc.

[0074] Early warning module: Make a prompt according to the alarm signal and contact the management personnel; The alarm signal includes a patient tension early warning signal, etc.

[0075] The database is used to store the data of each module and the historical data required for training the model.

[0076] In this embodiment, generating a recommended department for medical treatment according to the detection image and voice data includes:

[0077] Obtain the detection image and voice data;

[0078] Input the detection image into the medical record recognition table to obtain historical medical record labels and their corresponding medical record contents; The medical record recognition table is constructed by experts according to the historical medical record contents corresponding to the hospital's historical medical personnel; That is, the historical medical database corresponding to the hospital. Input the detection image into it, and the historical medical record can be obtained. Combining it with the problem description generated by the conversation with the robot, the recommended department for medical treatment can be obtained through the large model;

[0079] Integrate the voice data, historical medical record labels, and their corresponding medical record contents into recommended data;

[0080] Input the recommended data into the medical treatment recommendation model to obtain the recommended department for medical treatment; The medical treatment recommendation model is constructed through the large model.

[0081] In this embodiment, the medical treatment recommendation model is constructed through the large model, including:

[0082] Obtain several historical recommended data and their corresponding historical recommended departments for medical treatment;

[0083] Divide a number of historical recommendation data and their corresponding historical recommended departments for medical treatment into training data, validation data, and test data; perform data preprocessing on the training data, validation data, and test data to obtain a training set, a validation set, and a test set; the ratio among the training set, the test set, and the validation set is 7:2:1;

[0084] Select a large model as the basic model; the large model includes the T5 model, etc.;

[0085] Train the basic model with the training set, and adjust the learning rate and hyperparameters on the validation set to obtain a pre-trained model;

[0086] Verify the pre-trained model on the test set, and finally obtain a medical treatment recommendation model that takes the input recommendation data and outputs the recommended department for medical treatment.

[0087] In this embodiment, first obtain the historical medical records of the patient in the current hospital, and combine the voice content he said to obtain the recommended department for medical treatment through the pre-trained large model. In this way, the time required for the patient to find the service desk is reduced, and more accurate recommendations for departments for medical treatment can be provided, improving the accuracy of the medical treatment guidance system.

[0088] Generating the optimal guidance path according to the recommended department for medical treatment and the comprehensive monitoring video in this embodiment includes:

[0089] Obtain the personnel location corresponding to the patient, the comprehensive monitoring video, and the department location corresponding to the recommended department for medical treatment in real time; the comprehensive monitoring video refers to the monitoring videos collected by the cameras at various locations in the hospital;

[0090] Obtain a number of optional paths in the path table according to the personnel location and the department location; the path table is set by experts according to historical optional paths; that is, there are multiple paths that can communicate between the personnel location and the department location;

[0091] Generate a number of congestion indexes according to the comprehensive monitoring video; the congestion index refers to the congestion degree corresponding to the optional path;

[0092] After performing max-min normalization on a number of congestion indexes and the path distances corresponding to a number of optional paths, obtain the congestion weights and distance weights corresponding to a number of optional paths;

[0093] Select the optional path corresponding to the minimum value of the sum of the congestion weight and the distance weight as the optimal path.

[0094] Based on the location of the patient and the specific location of the recommended department, this embodiment generates multiple alternative paths. Among these alternative paths, factors such as path distance and congestion status are comprehensively weighed, and the optimal path is selected as the guiding path, enabling the patient to reach the department more quickly, thereby effectively improving the guiding accuracy and efficiency of the medical treatment guiding system.

[0095] In this embodiment, several congestion indices are generated based on the comprehensive monitoring videos, including:

[0096] Obtain several comprehensive monitoring videos; the comprehensive monitoring videos refer to the monitoring videos corresponding to the alternative paths;

[0097] Extract the population density RM, movement attenuation coefficient YJ, stagnation time ratio TT, and monitoring time from the comprehensive monitoring videos; the population density is expressed as the number of people per unit area, the movement attenuation coefficient is expressed as the ratio of the actual average movement speed of the crowd to the free flow speed; the stagnation time ratio is expressed as the proportion of the time when the crowd is in a stationary state within the statistical window time; these parameters can all be calculated from the comprehensive monitoring videos;

[0098] Through the formula Calculate the congestion index YZ i ; where, β1, β2, and β3 are adjustment coefficients, and β1, β2, and β3 are all greater than 0; the adjustment coefficients are adaptively adjusted according to the population density, monitoring time, and stagnation time ratio; in the path, the higher the population density, the more people there are, and the more difficult it is to pass through the path. Similarly, the smaller the movement attenuation coefficient and the larger the stagnation time ratio, the more difficult it is to pass through the path, so the congestion index increases accordingly.

[0099] In this embodiment, the adjustment coefficients are adaptively adjusted according to the population density, monitoring time, and stagnation time ratio, including:

[0100] Obtain the population density, monitoring time, and stagnation time ratio corresponding to several alternative paths; the adjustment coefficients corresponding to different monitoring times are different;

[0101] Integrate the population density, monitoring time, and stagnation time ratio corresponding to several alternative paths into adjustment coefficient data;

[0102] Input the adjustment coefficient data into the adjustment coefficient prediction model to obtain several adjustment coefficients corresponding to several alternative paths;

[0103] Among them, the adjustment coefficient prediction model is constructed through an artificial intelligence model, including:

[0104] Obtain several historical adjustment coefficient data and their corresponding several historical adjustment coefficients;

[0105] Divide a number of historical adjustment coefficient data and their corresponding historical adjustment coefficients into training data, verification data and test data; perform data preprocessing on the training data, verification data and test data to obtain a training set, a verification set and a test set; the ratio between the training set, the test set and the verification set is 7:2:1;

[0106] Select an artificial intelligence model as the basic model; the artificial intelligence model includes the BP model, etc.;

[0107] Train the basic model with the training set, and adjust the learning rate and hyperparameters on the validation set to obtain the pre-trained model;

[0108] By verifying the pre-trained model on the test set, the input adjustment coefficient data is finally obtained, and the output is an adjustment coefficient prediction model of several adjustment coefficients.

[0109] In this embodiment, the interactive strategy is generated according to the voice data and the body parameters and the alarm signal is generated, including:

[0110] Acquire voice data and body parameters; voice data includes intonation, body parameters include age and heart rate;

[0111] By formula Calculate the patient's tension coefficient; where DX represents the current heart rate, and JX represents the resting heart rate; YDB represents the pitch standard deviation, DP represents the jitter frequency, BYD represents the standard pitch value, and BDP represents the standard jitter frequency. The specific values ​​are set based on experience. In this embodiment, BYD is set to 100Hz and BDP is set to 5Hz; the resting heart rate and pitch standard deviation are calculated based on age and tone; different age groups correspond to different resting heart rates, so different resting heart rates need to be set according to the age of the patient. As the deviation between the current heart rate and the resting heart rate increases, and the pitch standard deviation and jitter frequency increase, it means that the patient is nervous, so the tension coefficient increases accordingly; the larger the standard deviation, the more drastic the tone fluctuation, and the more nervous the patient is;

[0112] When the tension coefficient is within the normal range, the normal range is set based on experience, and no operation is performed;

[0113] When the tension coefficient is within the tension range, the tension range is set according to experience, a patient tension warning signal is generated, and an interaction strategy is generated according to the tension coefficient.

[0114] When patients enter the hospital, they may feel nervous due to being in a hurry or even worried about their condition. At this time, when guiding the interaction, it is necessary to make adaptive changes based on the patient's state, such as using the soothing mode and changing the interaction frequency, so that the patient can appropriately relieve his or her nervousness and have a better experience of the medical process.

[0115] In this embodiment, the resting heart rate and the standard deviation of pitch are calculated according to age and intonation, including:

[0116] Obtain age and intonation; the intonation includes pitch values at N consecutive time points, that is, pitch values at N consecutive time points before the current time, where N is an integer, N>0, and the specific value is set according to experience;

[0117] The resting heart rate is equal to the average of several resting heart rates within the age range to which the age belongs;

[0118] The average pitch value YDP is obtained by taking the average of several pitch values;

[0119] Through the formula Calculate the standard deviation of pitch YDB; where YD n represents the pitch value at the nth time point between the current times; the more the several pitch values deviate from the corresponding average pitch value, the greater the intonation fluctuation, so the standard deviation of pitch increases accordingly.

[0120] In the process of guiding the patients, this embodiment will monitor the physical and mental states of the patients in real time; once it is detected that the patient shows a tense emotion, the guiding robot will automatically adjust the service mode, actively provide soothing services, and at the same time flexibly change the interaction frequency; through these measures, the tension of the patients is effectively relieved, providing a more considerate and user-friendly service experience for them, thereby improving the satisfaction of the patients.

[0121] Generating an interaction strategy according to the tension coefficient in this embodiment includes:

[0122] Obtain the tension coefficient and robot data; the robot data includes the interaction frequency and the interaction mode; the interaction mode includes the normal mode, the soothing mode, etc.;

[0123] When the tension coefficient is within the tension range, set the interaction mode to the soothing mode;

[0124] Through the formula Calculate the interaction frequency JP; where JP min represents the interaction frequency in the normal interaction mode, and the specific value is set according to experience. In this embodiment, JP minSet to 0.5 times per minute; ΔJP represents the maximum increase in the interaction frequency, and the specific value is set according to experience. Setting the maximum increase is to ensure that the interaction frequency can be controlled within a certain range. If no range is set, it may lead to an excessive interaction frequency of the robot, resulting in a situation where one interaction is not completed and another interaction arrives. In this embodiment, ΔJP is set to 3 times per minute; k represents the stress response coefficient, k > 0, and the specific value is set according to experience. In this embodiment, k is set to 1.5; the setting of k is to control the rate at which the interaction frequency increases with the stress coefficient. The larger k is, the more sensitive the interaction frequency is to the degree of stress; DJX represents the unit stress coefficient, and the specific value is set according to experience; when the patient becomes more and more nervous, the corresponding stress coefficient increases accordingly. At this time, the robot needs to adjust the interaction frequency to comfort the patient, and thus the interaction frequency increases accordingly.

[0125] Please refer to Figure 2 , another embodiment of the present application provides a method for guiding the visit of an intelligent robot in a hospital with multi-modal fusion, including:

[0126] S0: Obtain patient data, hospital comprehensive data, and robot data; the patient data includes a person ID, voice data, physical parameters, and detection images; the hospital comprehensive data includes several department IDs, department locations, and comprehensive monitoring videos;

[0127] S1: Generate a recommended department for the visit according to the detection image and voice data;

[0128] S2: Generate an optimal guiding path according to the recommended department for the visit and the comprehensive monitoring video;

[0129] S3: Generate an interaction strategy and generate an alarm signal according to the voice data and physical parameters;

[0130] S4: Make a prompt according to the alarm signal and contact the management personnel.

[0131] Some of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is obtained by software simulation of a large amount of collected data to get a formula closest to the actual situation; the preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0132] Working principle of this application: By obtaining data of patients seeking medical treatment, comprehensive hospital data, and robot data; generating recommended departments for medical treatment based on detection images and voice data; generating the optimal guiding path based on the recommended departments for medical treatment and comprehensive surveillance videos; generating interaction strategies and alarm signals based on voice data and physical parameters; giving prompts according to the alarm signals and contacting the management staff, combining the descriptions of patients seeking medical treatment with their historical medical records to quickly locate the departments for medical treatment, and constructing the optimal path based on the current location and the locations of the departments for medical treatment, reducing the time consumed by patients seeking medical treatment to ask for directions and enabling them to reach the departments for medical treatment as soon as possible; at the same time, adaptively adjusting the robot interaction strategy according to the status of patients seeking medical treatment, effectively alleviating the negative emotions of patients seeking medical treatment, improving user satisfaction, and avoiding the problems in the prior art that dynamic factors are not considered in path planning and the lack of perception and adaptive adjustment of patients' emotions, resulting in low accuracy and efficiency of the medical treatment guiding system and low satisfaction of patients seeking medical treatment.

[0133] The above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of this application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An intelligent robot medical guidance system within a hospital with multi-modal fusion, characterized in that, Including: A data acquisition module, a data analysis module, an early warning module, and a database; The data acquisition module: obtains patient data, hospital comprehensive data, and robot data through data acquisition devices; the patient data includes a person ID, voice data, physical parameters, and detection images; the hospital comprehensive data includes several department IDs, department locations, and comprehensive monitoring videos; The data analysis module: generates a recommended department for consultation based on the detection images and voice data; generates an optimal guiding path based on the recommended department for consultation and the comprehensive monitoring video; generates an interaction strategy and an alarm signal based on the voice data and physical parameters.

2. The intelligent robot visit guidance system within a hospital with multimodal fusion according to claim 1, characterized in that, Generating a recommended department for consultation based on the detection images and voice data includes: Obtaining the detection images and voice data; Inputting the detection images into a medical record recognition table to obtain historical medical record labels and their corresponding medical record contents; the medical record recognition table is constructed by experts based on the historical medical record contents of the hospital's historical patients; Integrating the voice data, historical medical record labels, and their corresponding medical record contents into recommended data; Inputting the recommended data into a consultation recommendation model to obtain a recommended department for consultation; the consultation recommendation model is constructed through a large model.

3. The intelligent robot visit guidance system within a hospital with multi-modal fusion according to claim 2, wherein, Constructing the consultation recommendation model through a large model includes: Obtaining a number of historical recommended data and their corresponding historical recommended departments for consultation; Dividing a number of historical recommended data and their corresponding historical recommended departments for consultation into training data, validation data, and test data; performing data preprocessing on the training data, validation data, and test data to obtain a training set, a validation set, and a test set; Selecting a large model as the basic model; Training the basic model with the training set and adjusting the learning rate and hyperparameters on the validation set to obtain a pre-trained model; Verifying the pre-trained model on the test set, and finally obtaining a consultation recommendation model that inputs recommended data and outputs the recommended department for consultation.

4. The intelligent robot visit guidance system within a hospital with multi-modal fusion according to claim 1, characterized in that, Generating an optimal guiding path based on the recommended department for consultation and the comprehensive monitoring video includes: Real-time obtaining the person's location corresponding to the patient, the comprehensive monitoring video, and the department location corresponding to the recommended department for consultation; Obtaining a number of optional paths in a path table based on the person's location and the department location; the path table is set by experts based on historical optional paths; Generating a number of congestion indices based on the comprehensive monitoring video; After performing maximum-minimum normalization on the number of congestion indices and the path distances corresponding to the number of optional paths, obtaining the congestion weights and distance weights corresponding to the number of optional paths; Selecting the optional path corresponding to the minimum value of the sum of the congestion weights and distance weights as the optimal path.

5. The intelligent robot visit guidance system in the hospital with multimodal fusion according to claim 4, characterized in that, Generating a number of congestion indices based on the comprehensive monitoring video includes: Obtaining a number of comprehensive monitoring videos; the comprehensive monitoring video refers to the monitoring video corresponding to the optional path; Extracting the crowd density RM, movement attenuation coefficient YJ, stagnation time ratio TT, and monitoring time from the comprehensive monitoring video; Calculate the congestion index YZ through the formula i ; where β1, β2, and β3 are adjustment coefficients, and β1, β2, and β3 are all greater than 0; the adjustment coefficients are adaptively adjusted according to the population density, monitoring time, and proportion of stagnation time.​ 6. The intelligent robot visit guidance system within a hospital with multi-modal fusion according to claim 5, characterized in that, The adjustment coefficient is adaptively adjusted according to the crowd density, monitoring time, and stagnation time ratio, including: Obtaining the crowd density, monitoring time, and stagnation time ratio corresponding to a number of optional paths; Integrate the population density, monitoring time, and stagnation time ratio corresponding to several optional paths into adjustment coefficient data; Input the adjustment coefficient data into the adjustment coefficient prediction model to obtain several adjustment coefficients corresponding to several optional paths; The adjustment coefficient prediction model is constructed through an artificial intelligence model, including: Obtain several historical adjustment coefficient data and their corresponding several historical adjustment coefficients; Divide several historical adjustment coefficient data and their corresponding several historical adjustment coefficients into training data, validation data, and test data; perform data preprocessing on the training data, validation data, and test data to obtain a training set, a validation set, and a test set; Select an artificial intelligence model as the basic model; Train the basic model through the training set and adjust the learning rate and hyperparameters on the validation set to obtain a pre-trained model; Verify the pre-trained model on the test set, and finally obtain an adjustment coefficient prediction model that inputs adjustment coefficient data and outputs several adjustment coefficients.

7. The intelligent robot visit guidance system within a hospital with multi-modal fusion according to claim 1, characterized in that, The generating of an interaction strategy and an alarm signal according to voice data and physical parameters includes: Obtain voice data and physical parameters; the voice data includes intonation, and the physical parameters include age and heart rate; Calculate the stress coefficient of the patient through the formula where DX represents the current heart rate, JX represents the resting heart rate; YDB represents the standard deviation of pitch, DP represents the jitter frequency, BYD represents the standard pitch value, and BDP represents the standard jitter frequency; the resting heart rate and the standard deviation of pitch are calculated according to age and intonation; When the tension coefficient is within the normal range, do nothing; When the tension coefficient is within the tense range, generate a patient tension warning signal and generate an interaction strategy according to the tension coefficient.

8. A multi-modal fusion intelligent robot medical treatment guidance system in a hospital according to claim 7, characterized in that, The resting heart rate and the standard deviation of the pitch are calculated according to age and intonation, including: Obtain age and intonation; the intonation includes pitch values at N consecutive time points; where N is an integer and N>0; The resting heart rate is equal to the average of several resting heart rates within the age range to which the age belongs; Obtain the average pitch value YDP by taking the average of several pitch values; Calculate the pitch standard deviation YDB through the formula ; where YD n represents the pitch value at the nth time point between the current times.

9. The intelligent robot visit guidance system in the hospital with multi-modal fusion according to claim 7, characterized in that The generating of an interaction strategy according to the tension coefficient includes: Obtain the tension coefficient and robot data; the robot data includes interaction frequency and interaction mode; the interaction mode includes a normal mode and a soothing mode; When the tension coefficient is within the tense range, set the interaction mode to the soothing mode; Calculate the interaction frequency JP through the formula where JP min represents the interaction frequency in the normal interaction mode; ΔJP represents the maximum growth rate of the interaction frequency; k represents the stress response coefficient, k > 0; and DJX represents the unit stress coefficient.

10. A method for guiding the visit of an intelligent robot in a hospital with multi-modal fusion, which is applied to a system for guiding the visit of an intelligent robot in a hospital with multi-modal fusion according to any one of claims 1-9, characterized in that, including: S0: Obtain patient visit data, hospital comprehensive data, and robot data; the patient visit data includes person ID, voice data, physical parameters, and detection images; the hospital comprehensive data includes several department IDs, department locations, and comprehensive monitoring videos; S1: Generate a recommended department for the visit according to the detection image and voice data; S2: Generate an optimal guiding path according to the recommended department for the visit and the comprehensive monitoring video; S3: Generate an interaction strategy and an alarm signal according to the voice data and physical parameters; S4: Make a prompt according to the alarm signal and contact the management staff.

Images