Method and apparatus for online consultation quality monitoring

Abstract

The application discloses an online consultation quality monitoring method and device, and relates to the technical field of internet medical treatment. A specific embodiment of the method comprises the following steps: in response to an online consultation quality monitoring instruction, obtaining conversation text between a doctor and a patient; extracting keywords from the conversation text, and determining a consultation type to which the conversation text belongs according to the extracted keywords and a preset scene classification rule; obtaining a diagnosis result determination rule of the consultation type, and performing diagnosis determination on the conversation text according to the diagnosis result determination rule, so as to perform online consultation quality monitoring. The embodiment classifies the most common scenes in actual online consultation, generalizes the scenes, formulates general rules to perform online consultation quality monitoring, thereby simplifying the online consultation quality monitoring process, improving the accuracy and efficiency of online consultation quality monitoring, and achieving the purpose of efficiently improving the effect of online consultation quality monitoring.

Description

Technical Field

[0001] This invention relates to the field of internet healthcare technology, and in particular to a method and apparatus for monitoring the quality of online consultations. Background Technology

[0002] Quality monitoring is a crucial aspect of internet healthcare, ensuring not only a positive user experience and accuracy in online consultations but also serving as an incentive for doctors. Traditional online consultation quality monitoring relies on random checks by experts or other doctors to manually review doctor-patient conversations and identify any issues. However, with the advancements in AI (Artificial Intelligence) technology, intelligent quality monitoring in online consultations now primarily utilizes deep learning, knowledge graphs, and natural language processing to process doctor-patient dialogues and detect problems, significantly improving the efficiency of quality monitoring.

[0003] However, in the process of realizing this invention, the inventors discovered that, since technologies such as natural language processing cannot fully understand human dialogue, and the question-and-answer dialogue between doctors and patients is not structured, there are some problems with semantic understanding when using artificial intelligence technology, resulting in a low accuracy rate of online consultation quality monitoring through AI technology. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for online consultation quality monitoring. Based on existing AI capabilities and related natural language processing technologies, it can classify the most common scenarios in actual online consultations, generalize the scenarios, and formulate general rules for online consultation quality monitoring. This simplifies the online consultation quality monitoring process, improves the accuracy and efficiency of online consultation quality monitoring, and ultimately achieves the goal of effectively improving the quality monitoring effect of online consultations.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for monitoring the quality of online consultations is provided, comprising: in response to an online consultation quality monitoring instruction, acquiring dialogue text between a doctor and a patient; extracting keywords from the dialogue text, and determining the consultation type to which the dialogue text belongs based on the extracted keywords and a preset scenario classification rule; acquiring a diagnostic result determination rule for the consultation type, and performing a diagnostic determination on the dialogue text based on the diagnostic result determination rule, so as to perform online consultation quality monitoring.

[0006] Optionally, the consultation type of the dialogue text is determined based on the extracted keywords and preset scenario classification rules, including: determining whether the extracted keywords meet preset image upload scenario rules; if the keywords meet the image upload scenario rules, determining that the consultation type of the dialogue text is image consultation; if the keywords do not meet the image upload scenario rules, determining whether the keywords meet preset explicit answer scenario rules; if the keywords meet the explicit answer scenario rules, determining that the consultation type of the dialogue text is explicit answer consultation; if the keywords do not meet the explicit answer scenario rules, determining that the consultation type of the dialogue text is other types of consultation.

[0007] Optionally, when the consultation type is an image consultation, the dialogue text is diagnosed and judged according to the diagnosis result judgment rules, including: judging that the doctor's diagnosis result corresponding to the dialogue text is correct.

[0008] Optionally, the keywords include a first keyword extracted from the doctor's script text and a second keyword extracted from the patient's script text; the keywords satisfying the explicit answer question-and-answer scenario rules include: the first keyword including a disease word and satisfying the disease word diagnosis rule, or the first keyword including a symptom word and satisfying the symptom word diagnosis rule; and, the dialogue text is diagnosed and judged according to the diagnosis result judgment rule, including: determining the patient's script text used to respond to the doctor's script text based on the doctor's script text containing the disease word or the symptom word; extracting answer keywords from the second keyword corresponding to the determined patient's script text; if the extracted answer keyword is an affirmative answer keyword, determining that the doctor's diagnosis result corresponding to the dialogue text is correct; if the extracted answer keyword is a negative answer keyword, determining that the doctor's diagnosis result corresponding to the dialogue text is incorrect.

[0009] Optionally, the scenario where the keywords satisfy the explicit answer question-and-answer scenario rules further includes: the first keyword includes question-and-answer option words and satisfies the question-and-answer option diagnosis rules, and the second keyword includes option results; and, the dialogue text is diagnosed and judged according to the diagnosis result judgment rules, including: determining the patient's script text used to reply to the doctor's script text based on the doctor's script text containing the question-and-answer option words; extracting option results from the second keyword corresponding to the determined patient's script text; and diagnosing and judging the dialogue text based on the option results and the pre-constructed quality monitoring knowledge graph.

[0010] Optionally, the keywords include a first keyword extracted from the doctor's text and a second keyword extracted from the patient's text; when the consultation type is another type of consultation, the dialogue text is diagnosed and judged according to the diagnosis result judgment rule, including: when the first keyword includes disease words and disease diagnosis indicators, and the second keyword includes the indicator value corresponding to the disease diagnosis indicator, the dialogue text is diagnosed and judged according to the judgment rule corresponding to the disease words, the disease diagnosis indicator, and the indicator value.

[0011] Optionally, the keywords include first keywords extracted from the doctor's script text; when the consultation type is other types of consultation, the dialogue text is diagnosed and judged according to the diagnosis result judgment rules, including: when the first keyword includes a disease word and a negative expression of the disease word, it is determined that the doctor did not make a diagnosis result related to the disease word.

[0012] Optionally, the keywords include first keywords extracted from the doctor's script text; when the consultation type is other types of consultation, the dialogue text is diagnosed and judged according to the diagnosis result judgment rules, including: when the first keywords include disease words and offline confirmed diagnosis keywords, the doctor's diagnosis result corresponding to the dialogue text is determined to be correct.

[0013] According to another aspect of the present invention, an apparatus for online consultation quality monitoring is provided, comprising: a dialogue text acquisition module, configured to acquire dialogue text between a doctor and a patient in response to an online consultation quality monitoring instruction; a consultation type determination module, configured to extract keywords from the dialogue text and determine the consultation type to which the dialogue text belongs based on the extracted keywords and a preset scenario classification rule; and a dialogue diagnosis determination module, configured to acquire a diagnosis result determination rule for the consultation type and perform a diagnosis determination on the dialogue text based on the diagnosis result determination rule, so as to perform online consultation quality monitoring.

[0014] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the online consultation quality monitoring method provided in the embodiments of the present invention.

[0015] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the online consultation quality monitoring method provided in the embodiments of the present invention.

[0016] One embodiment of the above invention has the following advantages or beneficial effects: By responding to an online consultation quality monitoring command, the technical solution of acquiring the dialogue text between the doctor and the patient; extracting keywords from the dialogue text and determining the consultation type of the dialogue text based on the extracted keywords and preset scenario classification rules; acquiring the diagnostic result judgment rules for that consultation type and diagnosing and judging the dialogue text according to the diagnostic result judgment rules, thus conducting online consultation quality monitoring, based on existing AI capabilities and related natural language processing technologies, classifies the most common scenarios in actual online consultations, and by generalizing the scenarios, formulates general rules for online consultation quality monitoring, thereby simplifying the online consultation quality monitoring process, improving the accuracy and efficiency of online consultation quality monitoring, and thus achieving the goal of effectively improving the online consultation quality monitoring effect.

[0017] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0018] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0019] Figure 1 This is a schematic diagram illustrating the main steps of an online consultation quality monitoring method according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram illustrating the implementation process of online consultation quality monitoring according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the main modules of an online consultation quality monitoring device according to an embodiment of the present invention;

[0022] Figure 4 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0023] Figure 5 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0024] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0026] In the description of the embodiments of the present invention, the technical terms involved and their definitions are as follows:

[0027] Regular expression matching: Regular expressions, also known as cyclic expressions, are text patterns that include ordinary characters (such as letters from a to z) and special characters (called "metacharacters"), and are a concept in computer science. Regular expressions use a single string to describe and match a series of strings that match a certain syntax rule. They are commonly used to search for and replace text that conforms to a specific pattern (rule).

[0028] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language.

[0029] Deep learning: Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans, and to recognize data such as text, images, and sound.

[0030] Knowledge Graph: A knowledge graph is a graph that displays the development process and structural relationships of knowledge, describes knowledge resources and their carriers, and mines, analyzes, constructs, draws and displays knowledge and the relationships between them.

[0031] Entity Relationship (NER) recognition, also known as entity relation recognition, refers to the task of extracting the implicit relationships between entities in a text during natural language processing. The extracted entity relationships can be expressed through different forms of language or other formats.

[0032] To address the technical problems existing in current technologies, this invention provides a method for monitoring the quality of online consultations. Based on existing AI capabilities and related natural language processing technologies, it categorizes and classifies the most common scenarios in actual online consultations. By generalizing these scenarios and establishing universal rules for each, the accuracy and effectiveness of online consultation quality monitoring are significantly improved.

[0033] Figure 1 This is a schematic diagram illustrating the main steps of an online consultation quality monitoring method according to an embodiment of the present invention. Figure 1 As shown, the online consultation quality monitoring method of this invention mainly includes the following steps S101 to S103.

[0034] Step S101: In response to the online consultation quality monitoring command, obtain the dialogue text between the doctor and the patient. After an online consultation ends, an online consultation quality monitoring command is triggered to monitor and evaluate the quality of the online consultation. Specifically, the dialogue text between the doctor and the patient generated during the online consultation can be obtained.

[0035] Step S102: Extract keywords from the dialogue text and determine the consultation type of the dialogue text based on the extracted keywords and preset scene classification rules. After obtaining the dialogue text between the doctor and the patient, scene classification is first performed based on the dialogue text to determine the consultation type. Specifically, keyword extraction from the dialogue text can be performed, for example, using entity relationship recognition technology.

[0036] According to one embodiment of the present invention, determining the consultation type of the dialogue text based on the extracted keywords and preset scenario classification rules includes: determining whether the extracted keywords satisfy preset image upload scenario rules; if the keywords satisfy the image upload scenario rules, determining that the consultation type of the dialogue text is image consultation; if the keywords do not satisfy the image upload scenario rules, determining whether the keywords satisfy preset explicit answer scenario rules; if the keywords satisfy the explicit answer scenario rules, determining that the consultation type of the dialogue text is explicit answer consultation; if the keywords do not satisfy the explicit answer scenario rules, determining that the consultation type of the dialogue text is other types of consultation.

[0037] Specifically, the process first determines whether the user uploaded an image during the online consultation. If so, and the following regular expression rule is met, the consultation type is classified as an image consultation: (based on | see | according to | through) (you)? .{0,6}? (of)? (examination report | report | photo | image | information | examination | result). This regular expression rule is the preset image upload scenario rule. When the extracted keywords meet this rule, the consultation type described in the dialogue text is an image consultation. If the extracted keywords do not meet the above regular expression rule, it is determined that no image was uploaded during the online consultation, and it can then be further determined whether the consultation type of the online consultation dialogue text is an explicit answer consultation. Similarly, regular expression matching can be used to check whether the dialogue text meets the preset explicit answer scenario rule. If it does, the consultation type of the dialogue text is determined to be an explicit answer consultation; otherwise, the consultation type of the dialogue text is determined to be another type of consultation.

[0038] According to one embodiment of the present invention, the keywords include a first keyword extracted from the doctor's script and a second keyword extracted from the patient's script. The keywords satisfying the explicit answer question-and-answer scenario rules include: the first keyword including a disease term and satisfying the disease term diagnosis rule; or the first keyword including a symptom term and satisfying the symptom term diagnosis rule; or, the first keyword including a question-and-answer option term and satisfying the question-and-answer option diagnosis rule, and the second keyword including the option result. Both disease terms and symptom terms can be pre-generated and stored in a database to facilitate the determination of the consultation type.

[0039] Specifically, the regular expression rules corresponding to the explicit question-and-answer scenario rules include, for example:

[0040] 1. Identify disease words from the doctor's text and satisfy the following regular expression (i.e., the first keyword includes the disease word and satisfies the disease word diagnosis rule): (have | have | exist | yes | confirmed | diagnosed) (yes | is)? .{2,40};

[0041] 2. Identify symptom words from the doctor's script that satisfy the following regular expression (i.e., the first keyword includes symptom words and satisfies the symptom word diagnosis rules): (have / have / exist / is / have / confirmed / diagnosed) (accompanied by / feeling)? .{2,60} (of / etc.)? (symptoms / conditions / manifestations);

[0042] 3. Identify question-and-answer option words from the doctor's text that satisfy the following regular expression, and the patient's response text includes the option result number (i.e., the first keyword includes question-and-answer option words and satisfies the question-and-answer option diagnosis rule, and the second keyword includes the option result): (Reply|Answer|Tell).{0,6}(Serial Number|Number).

[0043] If any one of the above three conditions is met, the consultation type of the dialogue text is determined to be an explicit response consultation. Otherwise, the consultation type of the dialogue text is determined to be another type of consultation.

[0044] Step S103: Obtain the diagnostic result determination rules for the consultation type, and perform diagnostic determination on the dialogue text according to the diagnostic result determination rules to monitor the quality of online consultations. After determining the consultation type to which the dialogue text belongs, the diagnostic result determination rules corresponding to that consultation type can be obtained, and diagnostic determination can be performed according to the diagnostic result determination rules.

[0045] According to one embodiment of the present invention, when the consultation type is an image-based consultation, the dialogue text is diagnosed and judged according to the diagnostic result judgment rules, including: determining that the doctor's diagnosis result corresponding to the dialogue text is correct. Since image files such as examination reports, images, or examination results basically contain preliminary diagnostic results, for image-based consultations, the doctor's diagnosis result can be directly determined to be correct. In this way, the use of existing AI technologies such as deep learning and knowledge graphs for online consultation quality monitoring can be avoided, thereby improving the accuracy and efficiency of online consultation quality monitoring.

[0046] According to another embodiment of the present invention, when the consultation type of the dialogue text is an explicit answer consultation, if it belongs to the first of the above three situations (the first keyword includes a disease word and meets the disease word diagnosis rule) or the second situation (the first keyword includes a symptom word and meets the symptom word diagnosis rule), then the dialogue text is diagnosed and judged according to the diagnosis result judgment rule. Specifically, this may include: determining the patient's script text used to reply to the doctor's script text based on the doctor's script text containing the disease word or the symptom word; extracting the answer keyword from the second keyword corresponding to the determined patient's script text; if the extracted answer keyword is an affirmative answer keyword, determining that the doctor's diagnosis result corresponding to the dialogue text is correct; if the extracted answer keyword is a negative answer keyword, determining that the doctor's diagnosis result corresponding to the dialogue text is incorrect.

[0047] Specifically, following the doctor's text containing disease or symptom terms, the system retrieves the patient's response text for the next sentence. If the patient's response text includes affirmative words such as "yes," "right," "correct," "have," "some," "all," "a little," "um," "exists," or "all," then the user is considered to have given an affirmative answer, and the doctor's diagnosis is considered correct. Otherwise, the doctor's diagnosis is considered incorrect.

[0048] According to another embodiment of the present invention, when the consultation type of the dialogue text is an explicit answer consultation, if it belongs to the third of the above three situations (the first keyword includes question-and-answer option words and satisfies the question-and-answer option diagnosis rule, and the second keyword includes option results), then the dialogue text is diagnosed and judged according to the diagnosis result judgment rule. Specifically, it may include: determining the patient's script text used to answer the doctor's script text based on the doctor's script text containing the question-and-answer option words; extracting the option results from the second keyword corresponding to the determined patient's script text; and diagnosing and judging the dialogue text based on the option results and the pre-constructed quality monitoring knowledge graph.

[0049] Specifically, following the doctor's text containing disease or symptom terms, the next sentence of the patient's response text is extracted. The system identifies the numbers in the user's response (note: if the user responds with multiple consecutive numbers, extract individual numbers. For example, if the user's response is text:125, the numbers 1, 2, and 5 are extracted), and matches them with the corresponding symptom descriptions from the doctor. These symptom descriptions are used as the basis for online consultation quality monitoring, further diagnosing and judging the dialogue text based on a pre-built quality monitoring knowledge graph. If the user's response does not contain numbers, the dialogue text is not considered for online consultation quality monitoring. It should be noted that this rule only applies when the doctor's maximum number is a single digit; if the doctor's maximum number is an even number, it is not used as the basis for online consultation quality monitoring. Doctors normally do not provide more than nine options; this rule is to prevent users from responding with two numbers without punctuation. The pre-built quality monitoring knowledge graph can use existing knowledge graphs built for online consultation quality monitoring; this invention does not limit this and will not elaborate further.

[0050] According to another embodiment of the present invention, the keywords include a first keyword extracted from the doctor's script text and a second keyword extracted from the patient's script text. When the consultation type of the dialogue text is not an image consultation or an explicit answer consultation, the present invention uniformly classifies it into other types of consultations. In the case of other types of consultations, the dialogue text is diagnosed and judged according to the diagnostic result judgment rules. Specifically, this may include: when the first keyword includes disease terms and disease diagnostic indicators, and the second keyword includes the indicator value corresponding to the disease diagnostic indicator, the dialogue text is diagnosed and judged according to the judgment rules corresponding to the disease terms, the disease diagnostic indicators, and the indicator values. Here, disease terms include, for example, overweight, hypertension, type 2 diabetes / gestational diabetes, etc., and the diagnosis of these diseases all require disease diagnostic indicators and the indicator values ​​corresponding to the disease diagnostic indicators.

[0051] For example, if the first keyword extracted from a doctor's dialogue (e.g., a doctor's question) includes the doctor's diagnostic terms "overweight" and diagnostic indicators such as "height and weight," "height and weight," and "height" and "weight," then three segments of the patient's dialogue following the doctor's statement can be extracted. If the second keyword extracted from the patient's dialogue contains two independent numbers, then the doctor's diagnosis corresponding to the dialogue is considered correct. Simultaneously, the patient's dialogue can be linked to the symptom term "BMI greater than 24.9" in the knowledge graph. Here, BMI stands for Body Mass Index. The criterion for determining the disease term "overweight" is "BMI greater than 24.9."

[0052] For example, if the first keyword extracted from a doctor's dialogue (e.g., a doctor's question) includes the doctor's diagnostic terms such as "hypertension" and "blood pressure," then three segments of the patient's dialogue following the doctor's statement can be extracted. If the second keyword extracted from the patient's dialogue contains two independent numbers, with the larger number greater than or equal to 140 and the smaller number greater than or equal to 90 (i.e., the judgment rule corresponding to the disease term "hypertension"), then the doctor's diagnosis in the dialogue text is determined to be correct. Simultaneously, the patient's dialogue can be linked to the symptom term "hypertension" in the knowledge graph via a chaining mechanism.

[0053] For example, if the first keyword extracted from the doctor's dialogue text (e.g., the doctor's questions) includes disease terms such as "type 2 diabetes / gestational diabetes" and diagnostic indicators such as "every other day blood glucose level" and "fasting blood glucose," then the patient's dialogue text following the doctor's statement is captured. If the second keyword extracted from the patient's dialogue text meets one of the following conditions, then the doctor's diagnosis corresponding to the dialogue text is determined to be correct:

[0054] 1. The patient's sentence contains the words "fasting", "before meals", and "before meals", and there is a number greater than 7.

[0055] 2. The patient's sentence contains the words "after meals" or "after eating," and one of them contains a number greater than 11;

[0056] 3. If the words "fasting", "before meals", or "before meals" are matched in the doctor's words, the next sentence of the patient's answer is captured. The patient's answer contains a number greater than 7.

[0057] 4. If the words "after a meal" or "after eating" are matched in the doctor's words, the next sentence of the patient's answer is captured. The patient's answer contains a number greater than 11.

[0058] Additionally, the patient's text can be linked to the symptom terms "every other day blood glucose > 11.1 or fasting blood glucose > 7" and "abnormal blood glucose" in the knowledge graph.

[0059] According to another embodiment of the present invention, the keywords include a first keyword extracted from the doctor's script text. When the consultation type is another type of consultation, the dialogue text is diagnosed and judged according to the diagnostic result determination rules. Specifically, this may include: if the first keyword includes a disease word and a negative expression of the disease word, determining that the doctor did not make a diagnostic result related to the disease word.

[0060] For example, if a disease term is identified in the first keyword extracted from the doctor's medical text, and any of the following regular expression rules are met, it is determined that the doctor did not make a diagnosis related to that disease term:

[0061] Regular expression 1: (exclude | do not consider | exclude).{0,20};

[0062] Regular expression 2: (consider | think).{2,30}(of)? (unlikely | impossible | unlikely).

[0063] According to another embodiment of the present invention, the keywords include a first keyword extracted from the doctor's script text. When the consultation type is another type of consultation, the dialogue text is diagnosed and judged according to the diagnostic result determination rules. Specifically, this may include: if the first keyword includes disease terms and offline confirmed diagnosis keywords, determining that the doctor's diagnosis result corresponding to the dialogue text is correct.

[0064] For example, if a disease word is identified in the first keyword extracted from the doctor's text and meets the following regular expression rule, then the doctor's diagnosis result corresponding to the dialogue text is determined to be correct: (See|According to)(you)? (Offline)(of)? (Hospital medical records|Medical records|Oral data|Diagnosis|Data).

[0065] Furthermore, for cases in embodiments of the present invention that cannot be handled through rule matching, they can be handled using existing AI technologies and related natural language processing technologies.

[0066] The online consultation quality monitoring method according to embodiments of the present invention can classify the most common scenarios in actual online consultations based on existing AI capabilities and related natural language processing technologies. By generalizing the scenarios, general rules are formulated for online consultation quality monitoring, thereby simplifying the online consultation quality monitoring process, improving the accuracy and efficiency of online consultation quality monitoring, and thus achieving the goal of effectively improving the online consultation quality monitoring effect.

[0067] Figure 2 This is a schematic diagram illustrating the implementation process of online consultation quality monitoring according to an embodiment of the present invention. In this embodiment, after an online consultation ends or after a set quality monitoring time has elapsed, an online consultation quality monitoring instruction is generated to monitor the quality of each online consultation. Specifically, firstly, the doctor-patient dialogue text of an online consultation is captured, including the doctor's and patient's text; then, keywords are extracted from the dialogue text; based on the extracted keywords, it is determined whether the patient uploaded an image during the online consultation; if so, the consultation type is an image consultation, and the doctor's diagnosis is correct; otherwise, it is further determined based on the extracted keywords whether the online consultation meets the explicit answer question-and-answer scenario rules; if so, the consultation type is an explicit answer consultation; otherwise, the consultation type is another type of consultation.

[0068] If the online consultation is a clear-answer consultation, then it is further determined which scenario of clear-answer consultation this online consultation belongs to. Scenario 1: The doctor's script text includes disease terms and meets the disease term diagnosis rules; Scenario 2: The doctor's script text includes symptom terms and meets the symptom term diagnosis rules; Scenario 3: The doctor's script text includes question-and-answer option terms and meets the question-and-answer option diagnosis rules, and the patient's script text includes option results. If the online consultation belongs to Scenario 1 and Scenario 2 of clear-answer consultation, then answer keywords are extracted from the patient's script text used to answer the doctor. If the answer keyword is an affirmative answer keyword, the doctor's diagnosis is correct; otherwise, the doctor's diagnosis is incorrect. If the online consultation belongs to Scenario 3 of clear-answer consultation, then option results are extracted from the patient's script text used to answer the doctor, and a diagnosis is made based on the option results and the quality monitoring knowledge graph.

[0069] If the online consultation is of another type, the diagnostic result determination rules corresponding to other types of consultations are further obtained, and a diagnosis is made based on the obtained diagnostic result determination rules. Specifically, if the online consultation dialogue text includes disease words and negative expressions for disease words, it is determined that the doctor did not make a diagnosis related to the disease words; if the online consultation dialogue text includes disease words and offline diagnostic keywords, it is determined that the doctor's diagnosis corresponding to the online consultation dialogue text is correct; if the doctor's dialogue text includes disease words, disease diagnostic indicators, and corresponding indicator values, the dialogue text is diagnosed based on the determination rules corresponding to the disease words, disease diagnostic indicators, and indicator values ​​to further determine whether the doctor's diagnosis is correct.

[0070] Figure 3 This is a schematic diagram of the main modules of an online consultation quality monitoring device according to an embodiment of the present invention. Figure 3 As shown, the online consultation quality monitoring device 300 of this embodiment mainly includes a dialogue text acquisition module 301, a consultation type determination module 302, and a dialogue diagnosis judgment module 303.

[0071] The dialogue text acquisition module 301 is used to acquire the dialogue text between doctors and patients in response to online consultation quality monitoring instructions;

[0072] The consultation type determination module 302 is used to extract keywords from the dialogue text and determine the consultation type of the dialogue text according to the extracted keywords and preset scene classification rules.

[0073] The dialogue diagnosis and judgment module 303 is used to obtain the diagnosis result judgment rules of the consultation type, and to perform diagnosis and judgment on the dialogue text according to the diagnosis result judgment rules, so as to monitor the quality of online consultation.

[0074] According to an embodiment of the present invention, the consultation type determination module 302 can also be used to: determine whether the extracted keywords meet the preset image upload scenario rules; if the keywords meet the image upload scenario rules, determine that the consultation type of the dialogue text is image consultation; if the keywords do not meet the image upload scenario rules, determine whether the keywords meet the preset explicit answer question-and-answer scenario rules; if the keywords meet the explicit answer question-and-answer scenario rules, determine that the consultation type of the dialogue text is explicit answer consultation; if the keywords do not meet the explicit answer question-and-answer scenario rules, determine that the consultation type of the dialogue text is other types of consultation.

[0075] According to another embodiment of the present invention, when the consultation type is an image consultation, the dialogue diagnosis determination module 303 can also be used to: determine that the doctor's diagnosis result corresponding to the dialogue text is correct.

[0076] According to another embodiment of the present invention, the keywords include a first keyword extracted from the doctor's script text and a second keyword extracted from the patient's script text. The keywords satisfying the explicit answer question-and-answer scenario rules include: the first keyword including a disease word and satisfying the disease word diagnosis rule, or the first keyword including a symptom word and satisfying the symptom word diagnosis rule. Furthermore, the dialogue diagnosis determination module 303 can also be used to: determine the patient's script text used to respond to the doctor's script text based on the doctor's script text containing the disease word or the symptom word; extract answer keywords from the second keyword corresponding to the determined patient's script text; determine that the doctor's diagnosis result corresponding to the dialogue text is correct if the extracted answer keyword is an affirmative answer keyword; and determine that the doctor's diagnosis result corresponding to the dialogue text is incorrect if the extracted answer keyword is a negative answer keyword.

[0077] According to another embodiment of the present invention, the case where the keyword satisfies the explicit answer question-and-answer scenario rule further includes: the first keyword includes question-and-answer option words and satisfies the question-and-answer option diagnosis rule, and the second keyword includes option results. Furthermore, the dialogue diagnosis and determination module 303 can also be used to: determine the patient's dialogue text used to respond to the doctor's dialogue text based on the doctor's dialogue text containing the question-and-answer option words; extract the option results from the second keyword corresponding to the determined patient's dialogue text; and diagnose and determine the dialogue text based on the option results and a pre-constructed quality monitoring knowledge graph.

[0078] According to another embodiment of the present invention, the keywords include a first keyword extracted from the doctor's text and a second keyword extracted from the patient's text. When the consultation type is another type of consultation, the dialogue diagnosis and determination module 303 can also be used to: diagnose and determine the dialogue text based on the determination rule corresponding to the disease word, the disease diagnosis indicator, and the indicator value, provided that the first keyword includes a disease term and a disease diagnosis indicator, and the second keyword includes an indicator value corresponding to the disease diagnosis indicator.

[0079] According to another embodiment of the present invention, the keywords include a first keyword extracted from the doctor's script text. In cases where the consultation type is another type of consultation, the dialogue diagnosis determination module 303 can also be used to: determine that the doctor did not make a diagnosis related to the disease word if the first keyword includes a disease word and a negative expression of the disease word.

[0080] According to another embodiment of the present invention, the keywords include a first keyword extracted from the doctor's script text. In cases where the consultation type is another type of consultation, the dialogue diagnosis determination module 303 can also be used to: determine that the doctor's diagnosis result corresponding to the dialogue text is correct if the first keyword includes disease terms and offline diagnosis keywords.

[0081] According to the technical solution of this invention, in response to an online consultation quality monitoring command, the system acquires the dialogue text between the doctor and the patient; extracts keywords from the dialogue text and determines the consultation type of the dialogue text based on the extracted keywords and preset scenario classification rules; acquires the diagnostic result judgment rules for the consultation type and performs diagnostic judgment on the dialogue text according to the diagnostic result judgment rules, thereby conducting online consultation quality monitoring. Based on existing AI capabilities and related natural language processing technologies, this system classifies the most common scenarios in actual online consultations, generalizes the scenarios, and formulates general rules for online consultation quality monitoring. This simplifies the online consultation quality monitoring process, improves the accuracy and efficiency of online consultation quality monitoring, and ultimately achieves the goal of significantly improving the effectiveness of online consultation quality monitoring.

[0082] Figure 4 An exemplary system architecture 400 is shown, which can be applied to the method or apparatus for online consultation quality monitoring according to embodiments of the present invention.

[0083] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, a network 404, and a server 405. Network 404 serves as the medium for providing communication links between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0084] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, and 403, such as online medical applications, online consultation applications, web browser applications, search applications, instant messaging tools, etc. (for example only).

[0085] Terminal devices 401, 402, and 403 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0086] Server 405 can be a server providing various services, such as a backend management server supporting websites browsed by users using terminal devices 401, 402, and 403 (for example only). The backend management server can retrieve data such as received online consultation quality monitoring instructions to obtain the dialogue text between doctors and patients; extract keywords from the dialogue text and determine the consultation type based on the extracted keywords and preset scenario classification rules; obtain the diagnostic result determination rules for the consultation type; perform diagnostic determination on the dialogue text according to the diagnostic result determination rules; and feed back the processing results (e.g., diagnostic determination results – for example only) to the terminal devices.

[0087] It should be noted that the online consultation quality monitoring method provided in this embodiment of the invention is generally executed by server 405, and correspondingly, the online consultation quality monitoring device is generally set in server 405.

[0088] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0089] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing terminal devices or servers of the present invention. Figure 5 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0090] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0091] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0092] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0093] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0095] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a dialogue text acquisition module, a consultation type determination module, and a dialogue diagnosis judgment module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, the dialogue text acquisition module can also be described as "a module for acquiring the dialogue text between doctors and patients in response to online consultation quality monitoring instructions."

[0096] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: in response to an online consultation quality monitoring instruction, acquiring dialogue text between a doctor and a patient; extracting keywords from the dialogue text and determining the consultation type of the dialogue text based on the extracted keywords and preset scenario classification rules; acquiring diagnostic result determination rules for the consultation type and performing diagnostic determination on the dialogue text based on the diagnostic result determination rules, thereby performing online consultation quality monitoring.

[0097] According to the technical solution of this invention, in response to an online consultation quality monitoring command, the system acquires the dialogue text between the doctor and the patient; extracts keywords from the dialogue text and determines the consultation type of the dialogue text based on the extracted keywords and preset scenario classification rules; acquires the diagnostic result judgment rules for the consultation type and performs diagnostic judgment on the dialogue text according to the diagnostic result judgment rules, thereby conducting online consultation quality monitoring. Based on existing AI capabilities and related natural language processing technologies, this system classifies the most common scenarios in actual online consultations, generalizes the scenarios, and formulates general rules for online consultation quality monitoring. This simplifies the online consultation quality monitoring process, improves the accuracy and efficiency of online consultation quality monitoring, and ultimately achieves the goal of significantly improving the effectiveness of online consultation quality monitoring.

[0098] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for monitoring the quality of online medical consultations, characterized in that, include: In response to online consultation quality monitoring commands, retrieve the dialogue text between doctors and patients; The dialogue text is subjected to keyword extraction, and the consultation type to which the dialogue text belongs is determined according to the extracted keywords and preset scene classification rules, including: determining whether the extracted keywords meet preset image upload scene rules; if the keywords meet the image upload scene rules, the consultation type to which the dialogue text belongs is determined to be image consultation; if the keywords do not meet the image upload scene rules, determining whether the keywords meet preset explicit answer question-and-answer scene rules; if the keywords meet the explicit answer question-and-answer scene rules, the consultation type to which the dialogue text belongs is explicit answer consultation; if the keywords do not meet the explicit answer question-and-answer scene rules, the consultation type to which the dialogue text belongs is determined to be other types of consultation. Obtain the diagnostic result determination rules for the consultation type, and perform diagnostic determination on the dialogue text according to the diagnostic result determination rules in order to monitor the quality of online consultations; In the case where the consultation type is an image consultation, the dialogue text is diagnosed and judged according to the diagnosis result judgment rule, including: judging that the doctor's diagnosis result corresponding to the dialogue text is correct; The keywords include a first keyword extracted from the doctor's script text and a second keyword extracted from the patient's script text. The keywords satisfying the explicit answer question-and-answer scenario rules include: the first keyword including a disease term and satisfying the disease term diagnosis rule, or the first keyword including a symptom term and satisfying the symptom term diagnosis rule. Furthermore, the dialogue text is diagnosed and judged according to the diagnosis result determination rules, including: determining the patient's script text used to respond to the doctor's script text based on the doctor's script text containing the disease term or the symptom term; extracting answer keywords from the second keyword corresponding to the determined patient's script text; determining that the doctor's diagnosis result corresponding to the dialogue text is correct if the extracted answer keyword is an affirmative answer keyword; and determining that the doctor's diagnosis result corresponding to the dialogue text is incorrect if the extracted answer keyword is a negative answer keyword. When the consultation type is other types of consultation, the dialogue text is diagnosed and judged according to the diagnosis result judgment rules, including: when the first keyword includes disease words and disease diagnosis indicators, and the second keyword includes the indicator value corresponding to the disease diagnosis indicator, the dialogue text is diagnosed and judged according to the judgment rules corresponding to the disease words, the disease diagnosis indicator and the indicator value.

2. The method according to claim 1, characterized in that, The conditions under which the keywords satisfy the explicit answer question-and-answer scenario rules also include: The first keyword includes question-and-answer option words and meets the question-and-answer option diagnosis rules, and the second keyword includes option results; Furthermore, the dialogue text is diagnosed and judged according to the diagnostic result judgment rules, including: Determine the patient's response text to the doctor's response text based on the doctor's response text containing the question and answer options; Extract the option results from the second keyword corresponding to the identified patient's verbal script; The dialogue text is diagnosed and judged based on the options and the pre-built quality monitoring knowledge graph.

3. The method according to claim 1, characterized in that, The keywords include the primary keywords extracted from the doctor's script text; When the consultation type is another type of consultation, the dialogue text is diagnosed and judged according to the diagnostic result judgment rules, including: If the first keyword includes a disease term and a negative expression of the disease term, it is determined that the doctor did not make a diagnosis related to the disease term.

4. The method according to claim 1, characterized in that, The keywords include the primary keywords extracted from the doctor's script text; When the consultation type is another type of consultation, the dialogue text is diagnosed and judged according to the diagnostic result judgment rules, including: If the first keyword includes disease terms and offline diagnosis keywords, the doctor's diagnosis result corresponding to the dialogue text is determined to be correct.

5. A device for monitoring the quality of online medical consultations, characterized in that, include: The dialogue text acquisition module is used to acquire the dialogue text between doctors and patients in response to online consultation quality monitoring commands; The consultation type determination module is used to extract keywords from the dialogue text and determine the consultation type of the dialogue text based on the extracted keywords and preset scene classification rules. The consultation type determination module is further configured to: determine whether the extracted keywords meet preset image upload scenario rules; if the keywords meet the image upload scenario rules, determine that the consultation type of the dialogue text is an image consultation; if the keywords do not meet the image upload scenario rules, determine whether the keywords meet preset explicit answer question-and-answer scenario rules; if the keywords meet the explicit answer question-and-answer scenario rules, determine that the consultation type of the dialogue text is an explicit answer consultation; if the keywords do not meet the explicit answer question-and-answer scenario rules, determine that the consultation type of the dialogue text is another type of consultation. The dialogue diagnosis and judgment module is used to obtain the diagnosis result judgment rules of the consultation type, and to perform diagnosis and judgment on the dialogue text according to the diagnosis result judgment rules, so as to monitor the quality of online consultation. In the case where the consultation type is an image consultation, the dialogue diagnosis and determination module is further used to: determine whether the doctor's diagnosis result corresponding to the dialogue text is correct; The keywords include a first keyword extracted from the doctor's script text and a second keyword extracted from the patient's script text; the keywords satisfying the explicit answer question-and-answer scenario rules include: the first keyword including a disease word and satisfying the disease word diagnosis rule, or the first keyword including a symptom word and satisfying the symptom word diagnosis rule; furthermore, the dialogue diagnosis determination module is used to: determine the patient's script text used to respond to the doctor's script text based on the doctor's script text containing the disease word or the symptom word; extract answer keywords from the second keyword corresponding to the determined patient's script text; if the extracted answer keyword is an affirmative answer keyword, determine that the doctor's diagnosis result corresponding to the dialogue text is correct; if the extracted answer keyword is a negative answer keyword, determine that the doctor's diagnosis result corresponding to the dialogue text is incorrect; When the consultation type is other types of consultation, the dialogue diagnosis and determination module is further configured to: when the first keyword includes disease words and disease diagnosis indicators, and the second keyword includes the indicator value corresponding to the disease diagnosis indicator, diagnose and determine the dialogue text according to the determination rule corresponding to the disease words, the disease diagnosis indicator and the indicator value.

6. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

7. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4.

Images