Intelligent hospital guide method and system based on symptom matching and department recommendation

Abstract

The invention discloses an intelligent hospital guide method and system based on symptom matching and department recommendation, and the method comprises the following steps: receiving a multi-modal symptom description inputted by a user, and carrying out the combined processing, thereby obtaining multi-modal symptom information; inputting the multi-modal symptom information into a multi-modal symptom deep learning model to obtain text key symptom features and voice key symptom features, and performing feature fusion to obtain comprehensive symptom feature representation; performing multi-dimensional matching according to the comprehensive symptom feature representation to generate a candidate department list; acquiring the real-time position of the user, inputting the real-time position of the user, the candidate department list and the voice key symptom features into a department recommendation model together, outputting a recommended department by calling a # imgabs0 # minimum spanning tree path algorithm, and generating a department information graph for the recommended department; and displaying department information, a doctor-seeing path and an appointment mode for the user through the department information atlas, thereby realizing intelligent hospital guide. According to the invention, the symptom information of the user is accurately identified and efficient matching with related departments is realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical technology, and in particular to an intelligent diagnosis guidance method and system based on symptom matching and department recommendation. Background Art

[0002] With the rapid development of information technology and artificial intelligence, smart healthcare has shown great potential in improving the efficiency of medical services and optimizing patients' medical experience. In the traditional medical model, patients often need to describe their symptoms in detail to obtain department recommendations during the medical treatment process. However, due to the differences in personal medical knowledge and the limitations of their expression ability, patients are often unable to accurately express their symptoms, resulting in inaccurate department recommendations and low medical efficiency. In addition, traditional medical guidance methods usually rely on manual or simple rule-based diagnostic systems, which lack flexibility and personalization and are difficult to cope with complex and changing medical scenarios.

[0003] With the development of multimodal perception technology, combined with natural language processing (NLP), speech recognition, computer vision and other technologies, the intelligent medical guidance system can better understand and process various input forms of patients, such as text, voice, images, etc., so as to achieve more accurate symptom recognition and department recommendation. Symptom matching methods based on multimodal information have become one of the hot topics in current intelligent medical guidance research.

[0004] However, most existing intelligent medical guidance systems focus on data processing in a single mode (such as voice recognition or text input), and the integration and matching of multimodal data still face certain challenges. In addition, the current system lacks comprehensive consideration of the real-time environment and the specific needs of patients in the process of department recommendation, making it difficult to provide accurate medical treatment paths and optimized department recommendations based on the patient's personalized symptom characteristics.

[0005] In order to further improve the intelligence and automation level of the intelligent medical guidance system, lower the threshold for users to seek medical treatment, and improve the efficiency and accuracy of medical treatment, there is an urgent need for an intelligent medical guidance method and system that integrates multimodal data analysis, deep learning and medical knowledge graphs. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides an intelligent medical guidance method and system based on symptom matching and department recommendation, which is used to solve the technical problems of inaccurate symptom recognition, inaccurate department recommendation, and unoptimized medical treatment path in the existing intelligent medical guidance technology, thereby accurately identifying the user's symptom information and achieving efficient matching with relevant departments, providing a more personalized, real-time and accurate medical treatment path.

[0007] To solve the above problems, the technical solution adopted by the present invention is as follows: An intelligent diagnosis guidance method based on symptom matching and department recommendation includes the following steps: Receiving a multimodal symptom description input by a user, and jointly processing the multimodal symptom description through natural language processing and speech recognition to obtain multimodal symptom information; Input the multimodal symptom information into the multimodal symptom deep learning model to obtain the text key symptom features and the voice key symptom features, and perform feature fusion to obtain the comprehensive symptom feature representation; Generate a list of candidate departments based on comprehensive symptom feature representation and multi-dimensional matching based on the medical knowledge graph; Get the user's real-time location and input it into the department recommendation model together with the candidate department list and key voice symptom features. The minimum spanning tree path algorithm outputs recommended departments and generates department information maps for the recommended departments; Through the department information map, users can see department information, treatment paths, and appointment methods to achieve intelligent guidance. Among them, the input of multimodal symptom descriptions is monitored in real time, and re-guidance is performed when changes occur; When multimodal symptom descriptions are processed jointly, include: The first modal symptom information is obtained by inputting the text symptom description into the symptom segmentation model for processing; The second modality symptom information is obtained by constructing a symptom evaluation dimension vector, a symptom intensity dimension vector, and a symptom excitation dimension vector for the speech symptom description; Combining the first modality symptom information with the second modality symptom information to obtain multimodal symptom information; Wherein, the multimodal symptom description includes: text symptom description and voice symptom description.

[0008] As a preferred embodiment of the present invention, the intelligent diagnosis guidance method further includes: Processing the image symptom description through visual recognition to obtain third modality symptom information; Combining the first modality symptom information, the second modality symptom information, and the third modality symptom information to obtain multimodal symptom information; Input the multimodal symptom information into the multimodal symptom deep learning model to obtain text key symptom features, voice key symptom features, image key symptom features, and perform feature fusion to obtain a comprehensive symptom feature representation; Wherein, the multimodal symptom description also includes: image symptom description.

[0009] As a preferred embodiment of the present invention, when input into the symptom segmentation model for processing, it includes: Based on the personalized word classification parameters, the parts of speech are classified and the text is segmented, and information is extracted from the content after the segmentation to obtain the first modal symptom information; Wherein, the personalized word classification parameters include: symptom word group, judgment word group; When obtaining second modality symptom information, including: According to the symptom arousal dimension vector and the symptom intensity dimension vector, the first symptom distance is obtained; According to the symptom arousal dimension vector and the symptom evaluation dimension vector, the second symptom distance is obtained; According to the symptom evaluation dimension vector and the symptom intensity dimension vector, the third symptom distance is obtained; According to the first symptom distance, the second symptom distance and the third symptom distance, the comprehensive symptom distance is obtained; The comprehensive symptom distance is associated with the three symptom distances respectively to obtain the multi-dimensional emotional space domain feature parameters as the second modal symptom information.

[0010] As a preferred embodiment of the present invention, when processing the image symptom description, it includes: Obtaining preset image vectors and text vectors to form image-text vector pairs, expanding the number of text vectors in the image-text vector pairs, and obtaining an image-text vector pair training data set; Input the image text vector pair training dataset into The language model is trained to obtain the pre-trained Language model; The image symptom description is input into the pre-trained In the model, after extracting the visual information of the image, the extracted visual information is mapped through a lightweight mapping network to obtain image vector; The image vectors are input to the pre-trained In the language model, it is connected with the corresponding text vector to output the third modality symptom information.

[0011] As a preferred embodiment of the present invention, when obtaining the key symptom features of speech and the key symptom features of image, it includes: The second modality symptom information and the third modality symptom information are input into the audio-visual deep feature encoding network layer of the multimodal symptom deep learning model, and the local spatial features of the second modality symptom information and the third modality symptom information are mined through the first convolutional neural network. The network extracts the time series features of the second modality symptom information and the third modality symptom information, and retains the long-sequence dependencies of the deep features; By connecting the first convolutional neural network and The network learns the deep spatiotemporal features associated with symptom recognition in the second modality symptom information and the third modality symptom information to the greatest extent.

[0012] As a preferred embodiment of the present invention, when obtaining the key symptom features of the text, it includes: The first modality symptom information is input into the multimodal symptom deep learning model. After encoding, they are respectively passed through the second convolutional neural network and The network performs feature extraction and concatenates the outputs of the two networks to obtain the key symptom features of the text; The second convolutional neural network extracts the hierarchical pattern of the sentence while maintaining a slight reduction in feature dimension. The network is forward and backward It consists of two parts, which are used to extract forward and reverse information, contextual information, and capture long sequence dependencies.

[0013] As a preferred embodiment of the present invention, when obtaining the comprehensive symptom characteristic expression, it includes: The key symptom features of text, voice and image are The function generates the weight matrix; Generate cross-modal fusion features of text key symptom features, speech key symptom features, and image key symptom features through an attention layer, wherein the attention layer uses a scaled dot product method to obtain attention; The cross-modal fusion features are passed The layer performs dimension scaling to make it consistent with the dimension of the weight matrix, and outputs the comprehensive symptom feature representation by performing a dot product with the weight matrix.

[0014] As a preferred embodiment of the present invention, when generating a candidate department list, it includes: Manually extract and integrate data from medical symptom databases and Excel resources to obtain structured data, and automatically obtain semi-structured data from medical websites using crawler parsers based on preset rules; The structured data and the semi-structured data are merged through knowledge merging, entity alignment, and entity disambiguation, and entities, relationships, and attributes are stored to obtain triple storage data; Construct a medical knowledge graph based on the entity information, relationship information, and attribute information in the triple storage data; Matching the comprehensive symptom feature representation with the entity information, relationship information and attribute information of each triple in the medical knowledge graph to obtain the triple with the highest matching degree; The relationship information in the triplet with the highest matching degree is obtained, and a candidate department list is generated according to the relationship information.

[0015] As a preferred embodiment of the present invention, when outputting the recommended department, it includes: The department recommendation model subscribes to the appointment, check-in, and reception messages of the triage system through the real-time data warehouse of triage and waiting, realizes real-time data push, and provides direct query access for downstream nodes through the interface; and generates a dictionary of medical project paths based on the candidate department list; The department recommendation model obtains relevant data fields from the real-time data warehouse of triage and waiting, and calls it based on the path dictionary of the medical project and the real-time location of the user. The minimum spanning tree path algorithm selects from all available edges in order of weight from small to large. If the selected edge does not form a loop with the existing edges, the selected edge is added to the minimum spanning tree to generate the optimal medical path, and the recommended department is output based on the optimal medical path.

[0016] An intelligent diagnosis guidance system based on symptom matching and department recommendation, including: Preprocessing unit: used for receiving multimodal symptom description input by the user, and jointly processing the multimodal symptom description through natural language processing and speech recognition to obtain multimodal symptom information; Feature extraction and fusion unit: used to input multimodal symptom information into the multimodal symptom deep learning model, obtain text key symptom features, voice key symptom features, and perform feature fusion to obtain comprehensive symptom feature representation; Matching unit: used to generate a list of candidate departments based on comprehensive symptom feature representation and multi-dimensional matching based on the medical field knowledge graph; Recommendation unit: used to obtain the user's real-time location and input it into the department recommendation model together with the candidate department list and key voice symptom features. The minimum spanning tree path algorithm outputs recommended departments and generates department information maps for the recommended departments; Guidance unit: used to display department information, consultation paths, and appointment methods to users through department information maps to achieve intelligent guidance; Among them, the input of multimodal symptom descriptions is monitored in real time, and re-guidance is performed when changes occur; When multimodal symptom descriptions are processed jointly, include: The first modal symptom information is obtained by inputting the text symptom description into the symptom segmentation model for processing; The second modality symptom information is obtained by constructing a symptom evaluation dimension vector, a symptom intensity dimension vector, and a symptom excitation dimension vector for the speech symptom description; Combining the first modality symptom information with the second modality symptom information to obtain multimodal symptom information; Wherein, the multimodal symptom description includes: text symptom description and voice symptom description.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) Multimodal information processing improves the accuracy of symptom description Through the joint processing of natural language processing, speech recognition and visual recognition technologies, it is possible to deeply understand the multimodal symptom descriptions (including text, voice and images) input by users, comprehensively obtain user symptom information, effectively solve the limitations of symptom expression caused by single modal input, and enable the system to adapt to a variety of symptom description methods.

[0018] (2) Accurate multimodal symptom feature extraction and fusion Through the multimodal symptom deep learning model, feature extraction and cross-modal fusion of symptom information in different modalities can be performed, which can comprehensively consider the key features of text, voice and image inputs to generate a more comprehensive comprehensive symptom feature representation, greatly improving the accuracy of symptom recognition and analysis.

[0019] (3) Multidimensional symptom matching based on medical knowledge graph By building a medical knowledge graph and matching comprehensive symptom features with triple information in the knowledge graph, we can achieve a deep association between symptom information and medical knowledge, accurately identify the potential causes of patients and match relevant departments. The application of medical knowledge graphs can effectively deal with complex symptoms and multiple causes, and provide users with highly matched candidate departments.

[0020] (4) Combining real-time location and dynamic recommendations to optimize the medical experience The system can combine the user's real-time location information to dynamically optimize department recommendations and treatment routes, and generate the optimal treatment route based on the minimum spanning tree path algorithm to help users quickly find the nearest and most suitable treatment department, greatly improving treatment efficiency and reducing patients' waiting time and unnecessary trips.

[0021] (5) Real-time update and dynamic response of intelligent medical guidance The system has the ability to monitor and dynamically update multimodal symptom descriptions in real time. When the user's symptom description changes, it can adjust the guidance results in real time, provide more flexible and accurate recommendations, adapt to changes in user needs, and ensure the timeliness and accuracy of the guidance service.

[0022] (6) Visual guidance service of department information map By generating a department information map, key content such as detailed information of recommended departments, treatment routes, appointment methods, etc. are intuitively displayed to users, helping users to quickly understand department details and treatment processes, reduce users' cognitive barriers to the complexity of the medical system, and optimize user experience.

[0023] (7) Multimodal feature mining based on deep learning By adopting deep convolutional neural networks and multimodal feature encoding networks, the system can efficiently extract local spatial features, time series features, and contextual correlation information of symptom features, and retain the long-sequence dependencies of deep features, comprehensively improving the representation ability of multimodal symptom features and the accuracy of symptom recognition.

[0024] (8) Efficient data fusion and knowledge integration By integrating structured and semi-structured medical data, the system builds a comprehensive medical knowledge graph that can perform entity alignment, disambiguation and relationship fusion on massive medical information, effectively improving the coverage and accuracy of knowledge and supporting efficient matching between symptoms and departments.

[0025] (9) Personalized medical guidance services improve user satisfaction The system combines the user's multimodal symptom description and personalized word classification parameters to provide more accurate guidance services based on the user's individual characteristics, avoiding the "one size fits all" problem in the traditional rule matching model, and greatly improving user satisfaction and medical efficiency.

[0026] (10) Improving the utilization rate of medical resources and triage efficiency Through intelligent, multimodal symptom recognition and precise recommendations, the system can quickly guide patients to the appropriate department, reduce invalid registrations and misdiagnosis rates, thereby effectively optimizing the utilization of medical resources, alleviating the triage pressure on medical institutions, and improving overall medical service efficiency.

[0027] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a step diagram of the intelligent diagnosis guidance method based on symptom matching and department recommendation provided by the present invention; Figure 2 It is a structural diagram of the sound-image deep feature encoding network layer of the multimodal symptom deep learning model provided by the present invention; Figure 3 It is a structural diagram of the text deep feature encoding network layer of the multimodal symptom deep learning model provided by the present invention.

[0029] Explanation of the reference numerals: 1. first convolution block; 2. first one-dimensional convolution layer; 3. first maximum pooling layer; 4. Activation function layer; 5. First Layer; 6, Network layer; 7, first fully connected layer; 8, first Activation function layer; 9. layer; 10, second fully connected layer; 11, second convolutional block; 12, second one-dimensional convolutional layer; 13, second Activation function layer; 14. First layer; 15, the third one-dimensional convolution layer; 16, the third Activation function layer; 17, second maximum pooling layer; 18, second Layer; 19, second Layer; 20, Layer; 21, third Layer; 22, third fully connected layer; 23, fourth Activation function layer; 24, fourth Layer; 25, Layer; 26, fifth Activation function layer; 27, fifth Layer; 28, Layer; 29, the fourth fully connected layer. DETAILED DESCRIPTION

[0030] The intelligent diagnosis guidance method based on symptom matching and department recommendation provided by the present invention is as follows: Figure 1 As shown, the following steps are included: Step S1: receiving a multimodal symptom description input by a user, and jointly processing the multimodal symptom description through natural language processing and speech recognition to obtain multimodal symptom information; Step S2: input the multimodal symptom information into the multimodal symptom deep learning model to obtain text key symptom features and speech key symptom features, and perform feature fusion to obtain a comprehensive symptom feature representation; Step S3: Generate a list of candidate departments based on the comprehensive symptom feature representation and multi-dimensional matching based on the medical field knowledge graph; Step S4: Obtain the user's real-time location and input it into the department recommendation model together with the candidate department list and the key speech symptom features, output the recommended department, and generate a department information map for the recommended department; Step S5: Display department information, medical treatment paths, and appointment methods to users through department information maps to achieve intelligent medical guidance; Among them, the input of multimodal symptom descriptions is monitored in real time, and when changes occur, the recommended department is re-output and the department information map is generated.

[0031] Preferably, step S1: receive a multimodal symptom description input by a user, and jointly process the multimodal symptom description through natural language processing, speech recognition, and visual recognition to obtain multimodal symptom information; step S2: input the multimodal symptom information into a multimodal symptom deep learning model to obtain text key symptom features, speech key symptom features, and image key symptom features, and perform feature fusion to obtain a comprehensive symptom feature representation.

[0032] In the above step S1, when the multimodal symptom description is jointly processed, it includes: Processing the text symptom description through natural language processing to obtain first modal symptom information; Processing the spoken symptom description through speech recognition to obtain second modality symptom information; Processing the image symptom description through visual recognition to obtain third modality symptom information; Combining the first modality symptom information, the second modality symptom information, and the third modality symptom information to obtain multimodal symptom information; Among them, multimodal symptom description includes: text symptom description, voice symptom description and image symptom description.

[0033] Furthermore, when processing the text symptom description, it includes: Input the text symptom description into the symptom segmentation model, classify the parts of speech and segment the text based on the personalized word classification parameters, extract information from the segmented content, and obtain the first modal symptom information; Among them, the personalized word classification parameters include: symptom word group, judgment word group; For symptom word groups, some synonyms are merged regularly and new symptom description words are added; For the judgment word group, it is divided into a negative judgment word group and a non-negative judgment word group, and the negative judgment word group is further divided into a pre-negative word group and a post-negative word group; commonly used expressions in outpatient medical record writing are obtained and added to the non-negative judgment word group.

[0034] Specifically, the word symptom cluster includes: abdominal distension, shortness of breath, fear of heat, fear of cold, abdominal pain, cold feet, fever, cough, cold hands, cough and wheezing, bitter taste in the mouth, dry mouth, and depression.

[0035] The non-negative judgment word group includes: maybe, possible, and definitely. The pre-negative word group includes: no, deny, never, no, and negate. The post-negative word group includes: uncommon, disappear, and not obvious.

[0036] Furthermore, when processing the voice symptom description, it includes: Construct symptom evaluation dimension vector, symptom intensity dimension vector, and symptom arousal dimension vector; According to the symptom excitation dimension vector and the symptom intensity dimension vector, the first symptom distance is obtained, as shown in Formula 1: (1); In the formula, is the first symptom distance, is the symptom excitation dimension vector, is the symptom intensity dimension vector; According to the symptom excitation dimension vector and the symptom evaluation dimension vector, the second symptom distance is obtained, as shown in Formula 2: (2); In the formula, is the second symptom distance, is the symptom evaluation dimension vector; According to the symptom evaluation dimension vector and the symptom intensity dimension vector, the third symptom distance is obtained, as shown in Formula 3: (3); In the formula, is the second symptom distance; According to the first symptom distance, the second symptom distance, and the third symptom distance, the comprehensive symptom distance is obtained, as shown in Formula 4: (4); In the formula, is the distance of comprehensive symptoms; The comprehensive symptom distance is associated with the three symptom distances respectively to obtain the multi-dimensional emotional space domain feature parameters as the second modal symptom information, as shown in Formula 5: (5); In the formula, is the characteristic parameter of the dimensional emotional space domain; Among them, when constructing the symptom evaluation dimension vector, it includes: Describe the speech symptoms Spectral analysis is performed to obtain a double-sided peak amplitude spectrum, and further a single-sided peak amplitude spectrum is obtained; the single-sided peak amplitude spectrum is subjected to energy normalization, frequency normalization, and smoothing to obtain a smoothed single-sided peak amplitude spectrum, and the average energy of the smoothed single-sided peak amplitude spectrum is obtained to obtain a symptom evaluation dimension vector; When constructing the symptom intensity dimension vector, include: The speech symptom description is decomposed into symptom approximation coefficients and symptom detail coefficients by wavelet decomposition; based on the symptom approximation coefficients and symptom detail coefficients, the subband signal is constructed by inverse discrete wavelet transform; the subband signal is subjected to discrete Fourier transform to obtain the scale peak of the subband signal, and the scale peaks are connected in series to obtain the symptom intensity dimension vector; When constructing the symptom-provoking dimension vector, include: The speech symptom description is framed by a windowing function, and the energy of each frame of the speech signal is obtained, so that the average energy of the speech symptom description is obtained as the symptom excitation dimension vector.

[0037] Furthermore, when processing the image symptom description, it includes: Obtaining preset image vectors and text vectors to form image-text vector pairs, expanding the number of text vectors in the image-text vector pairs, and obtaining an image-text vector pair training data set; Input the image text vector pair training dataset into The language model is trained to obtain the pre-trained Language model; Among them, the training objective is shown in Formula 6: (6); In the formula, are the training parameters that the model can learn, is the probability of generating a word vector, is the text vector, is the number of expanded text vectors, is the image vector, is the number of image vectors, is the logarithm of image-text vector pairs; Input the image symptom description into the pre-trained In the model, after extracting the visual information of the image, the extracted visual information is mapped through a lightweight mapping network to obtain image vector, as shown in Formula 7: (7); In the formula, for image vectors, For lightweight mapping network, For pre-trained Model, for image symptom description; Will image vectors are input to the pre-trained In the language model, it is connected with the corresponding text vector to output the third modality symptom information, as shown in Formula 8: (8); In the formula, For the third modality symptom information, is the corresponding text vector, is the number of corresponding text vectors.

[0038] like Figure 2 As shown, in the above step S2, when obtaining the key symptom features of speech and the key symptom features of image, it includes: The second modality symptom information and the third modality symptom information are input into the audio-visual deep feature encoding network layer of the multimodal symptom deep learning model, and the local spatial features of the second modality symptom information and the third modality symptom information are mined through the first convolutional neural network. The network extracts the time series features of the second modality symptom information and the third modality symptom information, and retains the long-sequence dependencies of the deep features; By connecting the first convolutional neural network and The network learns the deep spatiotemporal features associated with symptom recognition in the second modality symptom information and the third modality symptom information to the greatest extent; The first convolutional neural network includes three groups of first convolutional blocks 1, each group of first convolutional blocks 1 includes: a first one-dimensional convolutional layer 2, a first maximum pooling layer 3, Activation function layer 4; The first convolutional neural network passes through the first Layer 5 is connected in series to two The network layer 6 is then output by the first fully connected layer 7, and then passes through the first Activation function layer 8, After layer 9 and the second fully connected layer 10, the key symptom features of speech and image are output; like Figure 3 As shown, when obtaining the key symptom features of the text, it includes: The first modality symptom information is input into the multimodal symptom deep learning model. After encoding, they are respectively passed through the second convolutional neural network and The network performs feature extraction and concatenates the outputs of the two networks to obtain the key symptom features of the text; The second convolutional neural network extracts the hierarchical pattern of the sentence while maintaining a slight reduction in feature dimension. The network is forward and backward It consists of two parts, which are used to extract forward and reverse information, context information, and capture long sequence dependencies; The second convolutional neural network includes five groups of second convolutional blocks 11, each group of second convolutional blocks 11 includes: a second one-dimensional convolutional layer 12, a second Activation function layer 13, first Layer 14, the third one-dimensional convolution layer 15, the third Activation function layer 16, second maximum pooling layer 17, second Layer 18, second Layer 19; The network consists of two connected Layer 20, third Layer 21, third fully connected layer 22, fourth Activation function layer 23, fourth Layer 24; The second convolutional neural network and The output of the network goes through After the layer 25 is spliced, it passes through the fifth Activation function layer 26, fifth Layer 27, Layer 28 and the fourth fully connected layer 29 output text key symptom features.

[0039] In the above step S2, when obtaining the comprehensive symptom characteristic representation, it includes: The key symptom features of text, voice and image are The function generates the weight matrix; The cross-modal fusion features of text key symptom features, speech key symptom features, and image key symptom features are generated through the attention layer, where the attention layer uses the scaled dot product method to obtain attention, as shown in Formula 9 and Formula 10: (9); (10); In the formula, , , are trainable network parameters, The first fusion channel The output of the layer, For the second fusion channel The output of the layer, is the cross-modal fusion feature. is the attention layer; The cross-modal fusion features are passed The layer scales the dimension to match the dimension of the weight matrix and outputs the comprehensive symptom feature representation by performing a dot product with the weight matrix.

[0040] In the above step S3, when generating the candidate department list, it includes: Manually extract and integrate data from medical symptom databases and Excel resources to obtain structured data, and automatically obtain semi-structured data from medical websites using crawler parsers based on preset rules; The structured data and semi-structured data are integrated through knowledge merging, entity alignment, and entity disambiguation, and entities, relationships, and attributes are stored to obtain triple storage data; Construct a medical knowledge graph based on the entity information, relationship information, and attribute information in the triple storage data; Match the comprehensive symptom feature representation with the entity information, relationship information, and attribute information of each triple in the medical knowledge graph to obtain the triple with the highest matching degree; Obtain the relationship information in the triple with the highest matching degree, and generate a list of candidate departments based on the relationship information; Among them, the entity information includes the part corresponding to the disease, the attribute information includes the symptoms corresponding to the disease, and the relationship information includes the potential department corresponding to the disease.

[0041] In the above step S4, when outputting the recommended department, it includes: The department recommendation model subscribes to the appointment, check-in, and reception messages of the triage system through the real-time data warehouse of the triage waiting system to achieve real-time data push, and provides direct query access to downstream nodes through the interface; and generates a dictionary of medical project paths based on the candidate department list; The department recommendation model obtains relevant data fields from the real-time data warehouse of triage and waiting, and calls it based on the path dictionary of the medical project and the real-time location of the user. The minimum spanning tree path algorithm selects from all available edges in order of weight from small to large. If the selected edge does not form a loop with the existing edges, the selected edge is added to the minimum spanning tree to generate the optimal treatment path, and the recommended department is output based on the optimal treatment path; Among them, each node in the minimum spanning tree is the various links that patients need to go through, and the weight of the edge is the waiting time of each medical item plus the medical time plus the intensity of the multi-dimensional emotional space domain feature parameters; The consultation item path dictionary provides the department name, department specific location, clinic name, and clinic specific location; The data fields of the real-time data warehouse for triage and waiting include: message time, clinic number, registration serial number, registered doctor, process status (appointment, check-in, reception), status occurrence time, and appointment time.

[0042] In the above step S4, when generating a department information map for the recommended department, it includes: Obtain the category nodes of the recommended departments, which include: department information, treatment path, and appointment method; obtain the relationship between the category nodes and the weight of the relationship, and form a hierarchical tree structure with the category nodes to obtain the department information map; Use Docker containers to deploy the front-end and back-end separation framework, use MySQL database and Neo4j graph database on the back end to implement data addition, deletion, query and modification, and provide an API interface to the front end; receive back-end data through the front end to display the department information map.

[0043] Furthermore, when real-time monitoring of the input of multimodal symptom descriptions is performed, it includes: By deploying network detectors at the input nodes and introducing time windows to capture multimodal symptom descriptions in different time periods, as shown in Formula 11: (11); In the formula, Capture multimodal symptom descriptions in different time periods for the time window, are the deployment parameters of the network detector at the input node, is the association weight parameter between the time window and the multimodal symptom description, is the size of the time window, is the adaptive parameter of the time window, is the connection relationship between the first modal symptom information, the second modal symptom information, and the third modal symptom information in the time window; The multimodal symptom descriptions in different time periods captured by the time window are compared in real time. When inconsistency occurs, it is considered that the multimodal symptom description has changed.

[0044] The intelligent medical guidance system based on symptom matching and department recommendation provided by the present invention includes: a preprocessing unit, a feature extraction and fusion unit, a matching unit, a recommendation unit, and a medical guidance unit.

[0045] Preprocessing unit: used to receive the multimodal symptom description input by the user, and jointly process the multimodal symptom description through natural language processing and speech recognition to obtain multimodal symptom information.

[0046] Feature extraction and fusion unit: used to input multimodal symptom information into the multimodal symptom deep learning model, obtain text key symptom features, speech key symptom features, and perform feature fusion to obtain a comprehensive symptom feature representation.

[0047] Matching unit: used to generate a list of candidate departments based on comprehensive symptom feature representation and multi-dimensional matching based on the medical field knowledge graph.

[0048] Recommendation unit: used to obtain the user's real-time location, and input it into the department recommendation model together with the candidate department list and voice key symptom features, output the recommended department, and generate a department information map for the recommended department.

[0049] Guidance unit: used to display department information, treatment routes, and appointment methods to users through department information maps to achieve intelligent guidance.

[0050] Among them, the input of multimodal symptom descriptions is monitored in real time, and when changes occur, the recommended department is re-output and the department information map is generated.

[0051] Preferably, the preprocessing unit is used to receive a multimodal symptom description input by a user, and jointly process the multimodal symptom description through natural language processing, speech recognition, and visual recognition to obtain multimodal symptom information; the feature extraction and fusion unit is used to input the multimodal symptom information into a multimodal symptom deep learning model to obtain text key symptom features, speech key symptom features, image key symptom features, and perform feature fusion to obtain a comprehensive symptom feature representation.

[0052] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. An intelligent diagnosis guidance method based on symptom matching and department recommendation, characterized in that: The following steps are involved: Receiving a multimodal symptom description input by a user, and jointly processing the multimodal symptom description through natural language processing and speech recognition to obtain multimodal symptom information; Input the multimodal symptom information into the multimodal symptom deep learning model to obtain the text key symptom features and the voice key symptom features, and perform feature fusion to obtain the comprehensive symptom feature representation; Generate a list of candidate departments based on comprehensive symptom feature representation and multi-dimensional matching based on the medical knowledge graph; Get the user's real-time location and input it into the department recommendation model together with the candidate department list and key voice symptom features. The minimum spanning tree path algorithm outputs recommended departments and generates department information maps for the recommended departments; Through the department information map, users can see department information, treatment paths, and appointment methods to achieve intelligent guidance. Among them, the input of multimodal symptom descriptions is monitored in real time, and re-guidance is performed when changes occur; When multimodal symptom descriptions are processed jointly, include: The first modal symptom information is obtained by inputting the text symptom description into the symptom segmentation model for processing; The second modality symptom information is obtained by constructing a symptom evaluation dimension vector, a symptom intensity dimension vector, and a symptom excitation dimension vector for the speech symptom description; Combining the first modality symptom information with the second modality symptom information to obtain multimodal symptom information; Wherein, the multimodal symptom description includes: text symptom description and voice symptom description.

2. The intelligent diagnosis guidance method based on symptom matching and department recommendation according to claim 1, characterized in that: The intelligent diagnosis guidance method further includes: Processing the image symptom description through visual recognition to obtain third modality symptom information; Combining the first modality symptom information, the second modality symptom information, and the third modality symptom information to obtain multimodal symptom information; Input the multimodal symptom information into the multimodal symptom deep learning model to obtain text key symptom features, voice key symptom features, image key symptom features, and perform feature fusion to obtain a comprehensive symptom feature representation; Wherein, the multimodal symptom description also includes: image symptom description.

3. The intelligent diagnosis guidance method based on symptom matching and department recommendation according to claim 2 is characterized in that: When input into the symptom segmentation model for processing, it includes: Based on the personalized word classification parameters, the parts of speech are classified and the text is segmented, and information is extracted from the content after the segmentation to obtain the first modal symptom information; Wherein, the personalized word classification parameters include: symptom word group, judgment word group; When obtaining second modality symptom information, including: According to the symptom arousal dimension vector and the symptom intensity dimension vector, the first symptom distance is obtained; According to the symptom arousal dimension vector and the symptom evaluation dimension vector, the second symptom distance is obtained; According to the symptom evaluation dimension vector and the symptom intensity dimension vector, the third symptom distance is obtained; According to the first symptom distance, the second symptom distance and the third symptom distance, the comprehensive symptom distance is obtained; The comprehensive symptom distance is associated with the three symptom distances respectively to obtain the multi-dimensional emotional space domain feature parameters as the second modal symptom information.

4. The intelligent diagnosis guidance method based on symptom matching and department recommendation according to claim 2 is characterized in that: When processing the image symptom description, include: Obtaining preset image vectors and text vectors to form image-text vector pairs, expanding the number of text vectors in the image-text vector pairs, and obtaining an image-text vector pair training data set; Input the image text vector pair training dataset into The language model is trained to obtain the pre-trained Language model; The image symptom description is input into the pre-trained In the model, after extracting the visual information of the image, the extracted visual information is mapped through a lightweight mapping network to obtain image vector; The image vectors are input to the pre-trained In the language model, it is connected with the corresponding text vector to output the third modality symptom information.

5. The intelligent diagnosis guidance method based on symptom matching and department recommendation according to any one of claims 2 to 4, characterized in that: When obtaining the key symptom features of speech and the key symptom features of images, it includes: The second modality symptom information and the third modality symptom information are input into the audio-visual deep feature encoding network layer of the multimodal symptom deep learning model, and the local spatial features of the second modality symptom information and the third modality symptom information are mined through the first convolutional neural network. The network extracts the time series features of the second modality symptom information and the third modality symptom information, and retains the long-sequence dependencies of the deep features; By connecting the first convolutional neural network and The network learns the deep spatiotemporal features associated with symptom recognition in the second modality symptom information and the third modality symptom information to the greatest extent.

6. The intelligent diagnosis guidance method based on symptom matching and department recommendation according to claim 5 is characterized in that: When obtaining the key symptom features of the text, include: The first modality symptom information is input into the multimodal symptom deep learning model. After encoding, they are respectively passed through the second convolutional neural network and The network performs feature extraction and concatenates the outputs of the two networks to obtain the key symptom features of the text; The second convolutional neural network extracts the hierarchical pattern of the sentence while maintaining a slight reduction in feature dimension. The network is forward and backward It consists of two parts, which are used to extract forward and reverse information, contextual information, and capture long sequence dependencies.

7. The intelligent diagnosis guidance method based on symptom matching and department recommendation according to claim 6 is characterized in that: When a comprehensive symptom profile is obtained, it includes: The key symptom features of text, voice and image are The function generates the weight matrix; Generate cross-modal fusion features of text key symptom features, speech key symptom features, and image key symptom features through an attention layer, wherein the attention layer uses a scaled dot product method to obtain attention; The cross-modal fusion features are passed The layer performs dimension scaling to make it consistent with the dimension of the weight matrix, and outputs the comprehensive symptom feature representation by performing a dot product with the weight matrix.

8. The intelligent diagnosis guidance method based on symptom matching and department recommendation according to claim 1 or 7, characterized in that: When generating a list of candidate departments, include: Manually extract and integrate data from medical symptom databases and Excel resources to obtain structured data, and automatically obtain semi-structured data from medical websites using crawler parsers based on preset rules; The structured data and the semi-structured data are merged through knowledge merging, entity alignment, and entity disambiguation, and entities, relationships, and attributes are stored to obtain triple storage data; Construct a medical knowledge graph based on the entity information, relationship information, and attribute information in the triple storage data; Matching the comprehensive symptom feature representation with the entity information, relationship information and attribute information of each triple in the medical knowledge graph to obtain the triple with the highest matching degree; The relationship information in the triplet with the highest matching degree is obtained, and a candidate department list is generated according to the relationship information.

9. The intelligent diagnosis guidance method based on symptom matching and department recommendation according to claim 8, characterized in that: When outputting recommended departments, include: The department recommendation model subscribes to the appointment, check-in, and reception messages of the triage system through the real-time data warehouse of triage and waiting, realizes real-time data push, and provides direct query access for downstream nodes through the interface; and generates a dictionary of medical project paths based on the candidate department list; The department recommendation model obtains relevant data fields from the real-time data warehouse of triage and waiting, and calls it based on the path dictionary of the medical project and the real-time location of the user. The minimum spanning tree path algorithm selects from all available edges in order of weight from small to large. If the selected edge does not form a loop with the existing edges, the selected edge is added to the minimum spanning tree to generate the optimal medical path, and the recommended department is output based on the optimal medical path.

10. An intelligent medical guidance system based on symptom matching and department recommendation, characterized in that: include: Preprocessing unit: used for receiving multimodal symptom description input by the user, and jointly processing the multimodal symptom description through natural language processing and speech recognition to obtain multimodal symptom information; Feature extraction and fusion unit: used to input multimodal symptom information into the multimodal symptom deep learning model, obtain text key symptom features, voice key symptom features, and perform feature fusion to obtain comprehensive symptom feature representation; Matching unit: used to generate a list of candidate departments based on comprehensive symptom feature representation and multi-dimensional matching based on the medical field knowledge graph; Recommendation unit: used to obtain the user's real-time location and input it into the department recommendation model together with the candidate department list and key voice symptom features. The minimum spanning tree path algorithm outputs recommended departments and generates department information maps for the recommended departments; Guidance unit: used to display department information, consultation routes, and appointment methods to users through department information maps to achieve intelligent guidance; Among them, the input of multimodal symptom descriptions is monitored in real time, and re-guidance is performed when changes occur; When multimodal symptom descriptions are processed jointly, include: The first modal symptom information is obtained by inputting the text symptom description into the symptom segmentation model for processing; The second modality symptom information is obtained by constructing a symptom evaluation dimension vector, a symptom intensity dimension vector, and a symptom excitation dimension vector for the speech symptom description; Combining the first modality symptom information with the second modality symptom information to obtain multimodal symptom information; Wherein, the multimodal symptom description includes: text symptom description and voice symptom description.

Images