Multi-mode doctor-patient communication system and method, electronic equipment and medium

Abstract

The invention provides a multi-mode doctor-patient communication system and method, electronic equipment and a medium, and the system comprises a user layer which comprises a user interface operation module, an intelligent dialogue interaction module and a playing engine module, and is used for generating a corresponding instruction and / or obtaining personalized communication content according to user operation, and carrying out the multimedia display; the logic layer comprises an intelligent data analysis module, an intelligent content generation module and a real-time feedback updating module, and is used for receiving an instruction of the user layer, analyzing the instruction to obtain a dialogue response result, performing real-time monitoring analysis on a communication process according to the user operation data, the user basic data and the dialogue response result, and outputting a communication result; generating personalized communication content; and the data layer comprises a cloud storage module and a communication content management module and is used for performing data interaction with the user layer and the logic layer and storing user operation data and user basic data of the user layer and generated data of the logic layer. The multi-mode doctor-patient communication system improves personalization and efficiency of doctor-patient communication.

Description

Technical Field

[0001] This application belongs to the technical field of doctor-patient communication processing, and relates to a multimodal doctor-patient communication system, in particular to a multimodal doctor-patient communication system, method, electronic device and medium. Background Art

[0002] With the change of modern social lifestyle, the medical and health undertakings have become the key issues widely concerned by the society. However, the main reasons for the communication problems between doctors and patients lie in factors such as the knowledge gap, language communication barriers, and different cultural backgrounds between patients and medical staff. These barriers lead to poor communication between doctors and patients. Patients often cannot accurately and clearly express their own conditions, and the information transmission is incomplete or inaccurate, which in turn affects the diagnosis and treatment quality and effect. In addition, medical staff often use professional medical terms when communicating with patients, which patients are often difficult to understand, resulting in information misunderstanding or lack of effective feedback. Due to the symptoms of patients not being fully described, or medical staff being unable to obtain the complete medical history of patients, problems such as misdiagnosis and missed diagnosis often occur, delaying the treatment opportunity. At the same time, poor communication is also likely to cause doctor-patient conflicts and disputes, increasing the pressure on the medical industry.

[0003] Therefore, how to improve the effectiveness of doctor-patient communication and reduce the medical risks brought by communication barriers has become an important topic for improving medical quality and ensuring patient safety. Summary of the Invention

[0004] In view of the above-mentioned disadvantages of the prior art, the purpose of this application is to provide a multimodal doctor-patient communication system, method, electronic device and medium, which are used to solve the problem of low effectiveness of doctor-patient communication in the prior art.

[0005] In the first aspect, this application provides a multimodal doctor-patient communication system. The multimodal doctor-patient communication system includes: a user layer, including a user interface operation module, an intelligent dialogue interaction module and a playback engine module, which are used to generate corresponding instructions and / or obtain personalized communication content according to user operations, and perform multimedia display; a logic layer, including an intelligent data analysis module, an intelligent content generation module and a real-time feedback update module, which are used to receive the instructions of the user layer and parse the instructions to obtain dialogue response results, and perform real-time monitoring and analysis on the communication process according to user operation data, user basic data and dialogue response results to generate personalized communication content; a data layer, including a cloud storage module and a communication content management module, which are used to perform data interaction with the user layer and the logic layer, and store the user operation data, user basic data of the user layer and the generated data of the logic layer; wherein, the user operation data includes voice data and text data, the user basic data includes text data and image data, and the generated data includes voice data, text data and / or multimedia data.

[0006] In this application, a personalized multi-modal doctor-patient communication system is built using the user layer, logic layer, and data layer. According to multi-modal user information and medical knowledge, personalized communication content is generated and displayed in real time through the interaction and real-time analysis among various modules. Such a multi-modal doctor-patient communication system can solve the communication gap between doctors and patients, provide personalized services for users by combining multi-disciplinary, multi-modal knowledge and methods, and improve communication efficiency.

[0007] In one implementation of the first aspect, the intelligent dialogue interaction module interacts with the intelligent content generation module, and is used to generate corresponding voice instructions from the user's voice operations and send them to the intelligent content generation module.

[0008] In one implementation of the first aspect, the real-time feedback update module interacts with the playback engine module, and is used to generate communication suggestions in real time according to the analysis results of the intelligent data analysis module; and is also used to obtain the communication data of the user layer in real time to update the communication suggestions in real time; the communication data of the user layer includes communication duration, keywords, keyword occurrence frequency, and / or user emotional state.

[0009] In one implementation of the first aspect, the logic layer further includes: an intelligent content review module, which is used to monitor the communication process using a deep learning model and dynamically mask or replace the communication content according to a pre-stored keyword library; a remote control module, which is used to generate a control instruction according to the processing result of the intelligent content review module and remotely control the playback engine module for display.

[0010] In one implementation of the first aspect, the data layer further includes: an identity verification module, which interacts with the security and permission management module and is used to verify the user's identity to obtain an identity verification result; a security and permission management module, which is used to obtain the user's access permission according to the identity verification result.

[0011] In one implementation of the first aspect, the cloud storage module is used to classify and store data, and interact with the communication content management module in real time to synchronously update various types of data.

[0012] In one implementation of the first aspect, the data layer is further used to analyze and mine the user operation data, user basic data of the user layer and the generated data of the logic layer using machine learning algorithms and association rules, and feedback the analysis results to the real-time feedback update module.

[0013] Second aspect, the present application provides a multimodal doctor-patient communication method. The multimodal doctor-patient communication method includes: generating corresponding instructions and / or obtaining personalized communication content according to user operations, and performing multimedia display; receiving the instructions and parsing the instructions to obtain a dialogue response result; performing real-time monitoring and analysis on the communication process according to user operation data, user basic data, and the dialogue response result to generate personalized communication content; wherein, the user operation data includes voice data and text data, the user basic data includes text data and image data, and the personalized communication content includes voice data, text data, and / or multimedia data.

[0014] Third aspect, the present application provides an electronic device. The electronic device includes: a memory for storing a computer program; a processor for executing the computer program stored in the memory so that the electronic device executes the multimodal doctor-patient communication method as described in the second aspect.

[0015] Fourth aspect, the present application provides a computer-readable storage medium. A computer program is stored thereon, and when the program is executed by a processor, the multimodal doctor-patient communication method as described in the second aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A Shown is a schematic diagram of an application scenario of the multimodal doctor-patient communication method described in the present application.

[0017] Figure 1B Shown is a schematic diagram of the structure of the end-cloud interaction scenario in these implementation manners.

[0018] Figure 2 Shown is a schematic diagram of the structure of the multimodal doctor-patient communication system according to an embodiment of the present application.

[0019] Figure 3 Shown is a schematic diagram of the intelligent data analysis module according to an embodiment of the present application.

[0020] Figure 4 Shown is a schematic diagram of the user interface operation module according to an embodiment of the present application.

[0021] Figure 5 Shown is a schematic diagram of the multimodal doctor-patient communication system according to an embodiment of the present application.

[0022] Figure 6 Shown is a schematic flowchart of the multimodal doctor-patient communication method according to an embodiment of the present application.

[0023] Figure 7 Shown is a schematic diagram of the structure of the electronic device according to an embodiment of the present application.

[0024] Description of Component Labels

[0025] 1 Multimodal doctor-patient communication device

[0026] 11 Storage device

[0027] 12 Local processor

[0028] 13 Display terminal

[0029] 2 Terminal-cloud interaction system

[0030] 20 Terminal

[0031] 21 Cloud server

[0032] 100 Multimodal doctor-patient communication system

[0033] 110 User layer

[0034] 120 Logic layer

[0035] 130 Data layer

[0036] 700 Electronic device

[0037] 710 Memory

[0038] 720 Processor

[0039] 730 Display

[0040] Steps S11 - S13 Specific implementation manners

[0041] The following uses specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0042] It should be noted that in the embodiments of the present application, words such as "optionally" or "for example" represent examples, illustrations, or explanations. Any embodiment or design solution described as "optionally" or "for example" in the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, the use of words such as "optionally" or "for example" is intended to present related concepts in a specific manner.

[0043] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, or c may represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c may be single or multiple.

[0044] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner. Therefore, only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation may be arbitrarily changed, and the component layout type may also be more complex.

[0045] In the case of information asymmetry between medical staff and patients, due to the lack of standardized processes and effective communication skills, it is difficult for medical staff to convey information in an appropriate manner and unable to properly handle the emotional reactions of patients and their families, exacerbating the contradictions between the two sides. Due to the shortage of medical resources leading to insufficient psychological support for patients, it is difficult for medical staff to fully focus on the psychology of patients and their families, and their understanding of psychological support is limited, making patients feel extremely helpless when facing bad news.

[0046] At least for the above problems, the embodiments of the present application provide a multimodal doctor-patient communication system. The multimodal doctor-patient communication system includes: a user layer, including a user interface operation module, an intelligent dialogue interaction module, and a playback engine module, for generating corresponding instructions and / or obtaining personalized communication content according to user operations and performing multimedia display; a logic layer, including an intelligent data analysis module, an intelligent content generation module, and a real-time feedback update module, for receiving the instructions of the user layer and parsing the instructions to obtain dialogue response results, and performing real-time monitoring and analysis on the communication process according to user operation data, user basic data, and dialogue response results to generate personalized communication content; a data layer, including a cloud storage module and a communication content management module, for performing data interaction with the user layer and the logic layer, and storing the user operation data, user basic data of the user layer, and the generated data of the logic layer; where the user operation data includes voice data and text data, the user basic data includes text data and image data, and the generated data includes voice data, text data, and / or multimedia data.

[0047] In the embodiments of the present application, a personalized multi-modal doctor-patient communication system is built using a user layer, a logic layer, and a data layer. According to multi-modal user information and medical knowledge, personalized communication content is generated through the interaction and real-time analysis among various modules and displayed in real time. Such a multi-modal doctor-patient communication system can solve the communication gap between doctors and patients and provide personalized services for users by combining multi-disciplinary, multi-modal knowledge and methods.

[0048] Figure 1A Shown is a schematic diagram of an application scenario of the multi-modal doctor-patient communication method described in the present application. The multi-modal doctor-patient communication device 1 can be used to implement the multi-modal doctor-patient communication method provided in the embodiments of the present application, but the application scenarios of the multi-modal doctor-patient communication method provided in the embodiments of the present application are not limited to Figure 1A the multi-modal doctor-patient communication device 1 shown. As Figure 1A shown, the multi-modal doctor-patient communication device 1 includes a storage device 11, a local processor 12, and a display terminal 13. The multi-modal doctor-patient communication method provided in the embodiments of the present application can be applied to the local processor 12.

[0049] Among them, Figure 1A the local processor 12 in can be a single local processor or a local processor cluster or a cloud computing center composed of multiple local processors, etc., and there are no specific limitations here. Although Figure 1A only one storage device 11, one local processor 12, and one display terminal 13 are shown in, it should be understood that Figure 1A the examples in are only for understanding the solution, and the actual numbers of the local processor 12 and the display terminal 13 should be flexibly determined according to the actual situation.

[0050] In some other implementation manners, the multi-modal doctor-patient communication device 1 may also not include the display terminal 13, but only include a local processor 12 with a display function and a storage device 11. The multi-modal doctor-patient communication method provided in the embodiments of the present application can be applied to the local processor 12. The local processor 12 with a display function may include a tablet computer, a notebook computer, a palm computer, a mobile phone, a personal computer, and a voice interaction device, or may also be a face recognition device, etc., and there are no limitations here.

[0051] In still some other implementation manners, the multi-modal doctor-patient communication method described in the present application can be applied to an end-cloud interaction scenario. Figure 1B Shown is a schematic diagram of the structure of the end-cloud interaction scenario in these implementation manners. As Figure 1B shown, the end-cloud interaction system 2 includes a terminal 20 and a cloud server 21. Communication can be carried out between the terminal 20 and the cloud server 21, and the communication method is not limited to wired or wireless methods.

[0052] Among them, the terminal 20 can be mobile or fixed. For example, the terminal 20 can be a wireless terminal or a wired terminal. A wireless terminal can refer to a device with wireless transceiver functions, which can be deployed indoors, outdoors, and in medical laboratories. The terminal 20 can be a mobile phone, a tablet computer, a laptop computer, etc., which is not limited herein. The cloud server 21 can include one or more servers, or include one or more processing nodes, or include one or more virtual machines running on the server. The cloud server 21 can also be referred to as a server cluster, a management platform, a data processing center, etc., which is not limited in the embodiments of the present application.

[0053] Next, the technical solutions in the embodiments of the present application will be described in detail with reference to the accompanying drawings in the embodiments of the present application.

[0054] Figure 2 Shown is a schematic structural diagram of the multi-modal doctor-patient communication system described in the embodiments of the present application. As Figure 2 shown, the multi-modal doctor-patient communication system 100 includes a user layer 110, a logic layer 120, and a data layer 130.

[0055] The user layer 110 includes a user interface operation module, an intelligent dialogue interaction module, and a playback engine module, which are used to generate corresponding instructions and / or obtain personalized communication content according to user operations, and perform multimedia display.

[0056] The logic layer 120 includes an intelligent data analysis module, an intelligent content generation module, and a real-time feedback update module, which are used to receive the instructions of the user layer 110 and parse the instructions to obtain dialogue response results, and perform real-time monitoring and analysis on the communication process according to user operation data, user basic data, and the dialogue response results to generate personalized communication content.

[0057] The data layer 130 includes a cloud storage module and a communication content management module, which are used to perform data interaction with the user layer 110 and the logic layer 120, and store the user operation data, user basic data of the user layer 110, and the generated data of the logic layer 120.

[0058] Among them, the user operation data includes voice data and text data, the user basic data includes text data and image data, and the generated data includes voice data, text data, and / or multimedia data. Specifically, the user operation data and the user basic data are summarized as multi-modal data, which is used for the logic layer 120 to analyze and generate personalized communication content. For example, the user operation data includes data such as voice instructions, text instructions, and disease conditions input by the user, and the user basic data includes various types of data such as the user's diagnosis and treatment information, medical images, medical history records, medication information, and basic monitoring conditions.

[0059] In some possible implementations, the user layer 110 includes a user interface operation module, an intelligent conversation interaction module, and a playback engine module. The user interface operation module is used for the user to input operation instructions, such as asking about medication information or recovery time, etc. The intelligent conversation interaction module is used to generate a content request to the logic layer 120 according to the operation instructions input by the user. The playback engine module is used to receive instructions and play corresponding personalized communication content. The logic layer 120 includes an intelligent data analysis module, an intelligent content generation module, and a real-time feedback update module.

[0060] Figure 3 Shown is a schematic diagram of the intelligent data analysis module described in the embodiments of the present application. As Figure 3 shown, the intelligent data analysis module is used to perform data analysis based on user operation data, user basic data, and the dialogue response results to generate personalized communication content. The intelligent data analysis module includes a data collection and cleaning unit, a data storage and management unit, a data analysis and mining unit, a predictive analysis and model construction unit, a key indicator monitoring unit, and an optimization suggestion decision support unit. The data collection and cleaning unit is used to collect data generated by various data sources, including user operation data, user basic data, and medical-related data, and preprocess the data to handle missing values, duplicates, or abnormal data. The data analysis and mining unit identifies potential patterns in the user basic data and operation data through data cleaning, feature extraction, and association analysis, analyzes the association relationship between the user basic data and user operation data using association rule mining and machine learning algorithms, and feeds the analysis results back to the real-time feedback update module. The real-time feedback update module dynamically adjusts the personalized communication content according to the analysis results, and constructs a predictive model to provide data-driven decision support for doctor-patient communication.

[0061] The data layer 130 includes a cloud storage module and a communication content management module. The cloud storage module is used to store medical content, data, and resources, interact with the communication content management module and the intelligent data analysis module, and provide storage and retrieval functions for the communication content. The communication content management module is used for users to upload, edit, and manage medical content, and connect to the cloud storage module to store the medical content on the cloud server.

[0062] In some other possible implementations, Figure 4 Shown is a schematic diagram of the user interface operation module described in the embodiments of the present application. As Figure 4 shown, the user interface operation module includes a navigation bar, medical content management, data monitoring and analysis, device and configuration, user personal information and account management, and a help and support unit.

[0063] In an embodiment of the present application, the intelligent dialogue interaction module interacts with the intelligent content generation module, and is used to generate corresponding voice instructions from the user's voice operations and send them to the intelligent content generation module.

[0064] In some possible implementation manners, the intelligent dialogue interaction module interacts with the intelligent content generation module, and can convert the voice operations input by the user into corresponding voice instructions and send them to the intelligent content generation module, so as to generate personalized communication content for the user's voice operations.

[0065] In an embodiment of the present application, the real-time feedback update module interacts with the playback engine module, and is used to generate communication suggestions in real time according to the analysis results of the intelligent data analysis module; and is also used to obtain the communication data of the user layer 110 in real time to update the communication suggestions in real time; the communication data of the user layer 110 includes communication duration, keywords, keyword occurrence frequency, and / or user emotional state.

[0066] In some possible implementation manners, the real-time feedback update module interacts with the playback engine module, and is used to optimize communication scripts, diagnosis and treatment suggestions, and communication efficiency. Generate communication suggestions in real time according to the analysis results of the intelligent data analysis module, and is also used to obtain the communication data of the user layer 110 in real time to update the communication suggestions in real time. The communication data of the user layer 110 includes communication duration, keywords, the frequency of keyword occurrence, and user emotional state. Specifically, analyze the user's emotional state, cultural background, and condition, and push personalized communication content to medical staff in real time. The personalized communication content includes personalized script suggestions and diagnosis and treatment plans, as well as health suggestions provided to users. The optimization results of the real-time feedback update module are fed back in real time by using the playback engine module. At the same time, generate a monitoring report for the communication process, recording the communication duration, keywords, keyword occurrence frequency, and / or user emotional state.

[0067] In an embodiment of the present application, the logic layer 120 further includes an intelligent content review module and a remote control module. The intelligent content review module is used to monitor the communication process by using a deep learning model, and dynamically mask or replace the communication content according to a pre-stored keyword library. The remote control module is used to generate a control instruction according to the processing result of the intelligent content review module, and remotely control the playback engine module for display.

[0068] In some possible implementations, the logic layer 120 further includes an intelligent content review module and a remote control module. The intelligent content review module is used to use a deep learning model, such as a natural language processing module, to monitor the communication process in real time, perform keyword retrieval on negative emotion words, sensitive topic words, or medical academic misunderstanding words in the communication content according to a pre-stored keyword library, and perform dynamic shielding or replacement in combination with context analysis, and send a warning reminder to medical staff to ensure the smoothness of the communication environment. Specifically, the keyword library is dynamically updated according to the actual usage situation to improve the accuracy and adaptability of content review. The remote control module interacts with the playback engine module, and is used to generate a control instruction according to the content review result of the intelligent content review module, realize content update and scheduling management through a cloud server, connect to the user layer 110, and remotely control the playback engine module to display, allowing users to manage and monitor advertising devices through the remote control module.

[0069] In an embodiment of the present application, the data layer 130 further includes an identity authentication module. The identity authentication module interacts with the security and permission management module, and is used to authenticate the user identity to obtain an identity authentication result; the security and permission management module is used to obtain the user's access permission according to the identity authentication result.

[0070] In some possible implementations, the data layer 130 further includes an identity authentication module and a security and permission management module. The identity authentication module interacts with the security and permission management module, and is used to authenticate the user identity to ensure the access permission of authorized users to the system, and connect to the user interface and other key modules. The security and permission management module is used to obtain the user's access permission according to the identity authentication result to ensure that certain authorized users can access sensitive functions of the system.

[0071] In an embodiment of the present application, the cloud storage module is used to classify and store data, and interact with the communication content management module in real time to synchronously update various types of data.

[0072] In some possible implementations, the cloud storage module interacts with the communication content management module, and is used to classify and store data, and provide a data call function for the intelligent content generation module.

[0073] In an embodiment of the present application, the data layer 130 is further used to analyze and mine the user operation data, user basic data of the user layer and the generated data of the logic layer by using machine learning algorithms and association rules, and feedback the analysis results to the real-time feedback update module.

[0074] In some possible implementations, Figure 5It shows a schematic diagram of the multi-modal doctor-patient communication system described in the embodiments of the present application. As Figure 5 shown, the multi-modal doctor-patient communication system includes a user layer, a logic layer, and a data layer. The user layer includes a user interface operation module, an intelligent dialogue interaction module, and a playback engine module, which are used to generate corresponding instructions and / or obtain personalized communication content according to user operations and perform multimedia display.

[0075] The logic layer includes an intelligent data analysis module, an intelligent content generation module, a real-time feedback update module, an intelligent content review module, and a remote control module. It is used to receive the instructions from the user layer and parse the instructions to obtain the dialogue response results, and perform real-time monitoring and analysis on the communication process according to the user operation data, user basic data, and the dialogue response results to generate personalized communication content.

[0076] The data layer includes a cloud storage module, a communication content management module, an identity authentication module, and a security and permission management module. It is used to perform data interaction with the user layer and the logic layer, and store the user operation data, user basic data of the user layer, and the generated data of the logic layer.

[0077] The playback engine module interacts with the real-time feedback update module and is used to generate communication suggestions in real time according to the analysis results of the intelligent data analysis module. The intelligent dialogue interaction module interacts with the intelligent content generation module and is used to generate corresponding voice instructions from the user's voice operations and send them to the intelligent content generation module. The intelligent content review module interacts with the intelligent content generation module and is used to monitor the communication process using a deep learning model and perform dynamic shielding or replacement on the communication content according to the pre-stored keyword library. The remote control module interacts with the playback engine module and is used to generate control instructions according to the processing results of the intelligent content review module and remotely control the playback engine module for display. The cloud storage module interacts with the intelligent content generation module and is used to provide data support for the intelligent content generation module. The cloud storage module also interacts with the communication content management module and is used to classify and store data content. The identity authentication module interacts with the security and permission management module and is used to perform permission verification and update, as well as permission allocation for authorized users.

[0078] It should be noted that the above is only a possible implementation manner of the embodiments of the present application, and the specific interaction manner between modules is not limited thereto.

[0079] The following embodiments of the present application provide a multi-modal doctor-patient communication method, which can be implemented, for example, by Figure 1A the local processor 12 shown or Figure 1B the cloud server 21 shown. Figure 6It is a schematic flowchart showing the multi-modal doctor-patient communication method described in the embodiments of the present application. As Figure 6 shown, the multi-modal doctor-patient communication method includes steps S11 to S13.

[0080] Step S11: Generate corresponding instructions and / or obtain personalized communication content according to user operations, and perform multimedia display.

[0081] Step S12: Receive the instructions and parse the instructions to obtain a dialogue response result.

[0082] Step S13: Perform real-time monitoring and analysis on the communication process according to user operation data, user basic data, and the dialogue response result to generate personalized communication content.

[0083] Among them, the user operation data includes voice data and text data, the user basic data includes text data and image data, and the personalized communication content includes voice data, text data, and / or multimedia data.

[0084] In the embodiments of the present application, the communication process is monitored and analyzed in real time according to user operation data, user basic data, and the dialogue response result to generate personalized communication content. According to multi-modal user information and medical knowledge, personalized communication content is generated and displayed in real time by using the interaction and real-time analysis among various modules. Such a multi-modal doctor-patient communication method can solve the communication gap between doctors and patients and provide personalized services for users by combining multi-disciplinary, multi-modal knowledge and methods.

[0085] In several embodiments provided by the present application, it should be understood that the disclosed system, device, or method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules / units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of devices or modules or units can be in electrical, mechanical, or other forms.

[0086] The modules / units described as separate components may or may not be physically separated, and the components shown as modules / units may or may not be physical modules, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected according to actual needs to achieve the objectives of the embodiments of this application. For example, in the various embodiments of this application, the functional modules / units can be integrated in one processing module, or each module / unit can exist physically alone, or two or more modules / units can be integrated in one module / unit.

[0087] Those of ordinary skill in the art should also be able to further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0088] The embodiments of this application also provide an electronic device. Figure 7 The structural schematic diagram of the electronic device 700 described in the embodiments of this application is shown. As Figure 7 shown, in this embodiment, the electronic device 700 includes a memory 710 and a processor 720.

[0089] The memory 710 is used to store computer programs; preferably, the memory 710 includes: various media such as ROM, RAM, magnetic disks, USB flash drives, memory cards, or optical discs that can store program codes.

[0090] Specifically, the memory 710 may include a computer system readable medium in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device 700 may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 710 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present application. It can be understood that the memory 710 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be read-only memory (ROM, Read Only Memory), programmable read-only memory (PROM, Programmable Read-Only Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM, Static Random Access Memory), synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory). The memory described in the embodiments of the present invention is intended to include but not limited to these and any other suitable categories of memory.

[0091] The processor 720 is connected to the memory 710 and is configured to execute the computer program stored in the memory 710, so that the electronic device 700 executes the multi-modal doctor-patient communication method described in any embodiment of the present application.

[0092] Optionally, the processor 720 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0093] Optionally, the electronic device 700 in this embodiment may further include a display 730. The display 730 is communicatively connected to the memory 710 and the processor 720, and is configured to display a relevant graphical user interface (GUI) interaction interface of the multi-modal doctor-patient communication method described in the embodiments of the present application.

[0094] The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the multi-modal doctor-patient communication method described in any embodiment of the present application.

[0095] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, an application running on a computing device and the computing device can both be components. One or more components may reside in a process and / or an execution thread, and a component may be located on one computer and / or distributed between two or more computers. In addition, these components may execute from various computer-readable media storing various data structures. A component may communicate, for example, through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as data interacting with other systems through signals via the Internet).

[0096] The descriptions of the processes or structures corresponding to the above respective drawings each have their own focuses. For parts not detailed in a certain process or structure, reference may be made to the relevant descriptions of other processes or structures.

[0097] The above embodiments are only illustrative of the principles and effects of the present application, and are not intended to limit the present application. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed in the present application should still be covered by the claims of the present application.

Claims

1. A multimodal doctor-patient communication system, characterized in that: include: The user layer includes a user interface operation module, an intelligent dialogue interaction module, and a playback engine module, which are used to generate corresponding instructions and / or obtain personalized communication content according to user operations, and perform multimedia display; The logic layer includes an intelligent data analysis module, an intelligent content generation module and a real-time feedback update module, which are used to receive instructions from the user layer and parse the instructions to obtain dialogue response results, and to monitor and analyze the communication process in real time according to user operation data, user basic data and the dialogue response results to generate personalized communication content; The data layer includes a cloud storage module and a communication content management module, which is used to interact with the user layer and the logic layer, and store user operation data and user basic data of the user layer and generated data of the logic layer; The user operation data includes voice data and text data, the user basic data includes text data and image data, and the generated data includes voice data, text data and / or multimedia data.

2. The multimodal doctor-patient communication system according to claim 1, characterized in that: The intelligent dialogue interaction module interacts with the intelligent content generation module to generate corresponding voice instructions based on the user's voice operation and send them to the intelligent content generation module.

3. The multimodal doctor-patient communication system according to claim 1, characterized in that: The real-time feedback update module interacts with the playback engine module to generate communication suggestions in real time according to the analysis results of the intelligent data analysis module; as well as It is also used to obtain user-level communication data in real time to update the communication suggestions in real time; The communication data of the user layer includes communication duration, keywords, keyword occurrence frequency and / or user emotional state.

4. The multimodal doctor-patient communication system according to claim 1, characterized in that: The logic layer also includes: Intelligent content review module, which uses deep learning models to monitor the communication process and dynamically block or replace communication content based on pre-stored keyword lexicon; The remote control module is used to generate control instructions according to the processing results of the intelligent content review module, and remotely control the playback engine module for display.

5. The multimodal doctor-patient communication system according to claim 1, characterized in that: The data layer also includes: The identity authentication module interacts with the security and authority management module to verify the user's identity and obtain the identity authentication result; The security and authority management module is used to obtain the user's access rights according to the identity authentication result.

6. The multimodal doctor-patient communication system according to claim 1, characterized in that: The cloud storage module is used to classify and store data, and interact with the communication content management module in real time to synchronously update various types of data.

7. The multimodal doctor-patient communication system according to claim 1, characterized in that: The data layer is also used to analyze and mine the user operation data of the user layer, the user basic data and the generated data of the logic layer using machine learning algorithms and association rules, and feed back the analysis results to the real-time feedback update module.

8. A multimodal doctor-patient communication method, characterized in that: include: Generate corresponding instructions and / or obtain personalized communication content according to user operations, and perform multimedia display; receiving the instruction and parsing the instruction to obtain a dialogue response result; Performing real-time monitoring and analysis on the communication process based on the user operation data, user basic data and the dialogue response results to generate personalized communication content; The user operation data includes voice data and text data, the user basic data includes text data and image data, and the personalized communication content includes voice data, text data and / or multimedia data.

9. An electronic device, characterized in that: The electronic device comprises: Memory for storing computer programs; A processor, wherein the processor is used to execute the computer program stored in the memory so that the electronic device executes the multimodal doctor-patient communication method as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multimodal doctor-patient communication method described in claim 8 is implemented.

Images