Medical question and answer risk control method and device

Abstract

The invention provides a medical question and answer risk control method and device, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the legality detection of the medical question content inputted by a user; when the legality detection result is passed, generating current answer content corresponding to the medical question content by using a question and answer large model, and performing streaming output of the current answer content; according to the continuously iteratively updated medical risk control knowledge base, performing medical risk control detection on the current answer content in real time by using a preset auditing model to obtain a current risk control detection result; and filtering and outputting the current answer content according to the current risk control detection result. According to the method, wrong medical information output caused by AI model defects can be effectively reduced, and the accuracy, safety and reliability of medical consultation and suggestions are ensured, so that the content safety risk problem of AI medical products is solved, and it is ensured that medical contents generated by AI conform to laws and regulations and medical professional standards.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a medical question-and-answer risk control method and device. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the application of natural language processing (NLP) in healthcare is becoming increasingly widespread. AI (artificial intelligence) medical products have gradually penetrated multiple aspects of medical services, including online consultations, auxiliary diagnosis, and health consultations. These products are often built on large-scale pre-trained language models and can handle diverse interactive needs in medical scenarios, including symptom descriptions, disease consultations, and medication guidance. However, due to the high level of professionalism and strict safety requirements of the medical industry, any content output by AI systems that is erroneous, misleading, or inconsistent with medical standards can have serious consequences for the patient's health decisions and even endanger their lives.

[0003] Currently, some AI medical products are still in the early stages of content security control. Some systems rely entirely on the output of general-purpose large models, using only simple prompts to guide the model to generate expected content. However, this approach struggles to effectively mitigate inherent model artifacts and semantic biases, and lacks deep integration and verification of medical expertise. Furthermore, some products employ rule-based keyword filtering, such as blacklisting high-risk terms like "cure" and "absolutely effective." However, this approach is limited by rules and coverage, making it incapable of handling complex medical semantic logic. For example, whether information such as "a certain drug is contraindicated for pregnant women" poses a risk requires contextual understanding. This statement alone may not apply to all scenarios, and traditional rule engines struggle to accurately identify its semantic intent, leading to misjudgments or omissions. Therefore, in the current technological landscape, building a scientific, rigorous, and implementable medical content risk control system has become a critical challenge for AI medical products. Summary of the Invention

[0004] The purpose of this invention is to provide a medical question-and-answer risk control method and device to address the content security risk issues of AI medical products and ensure that AI-generated medical content complies with laws, regulations, and medical professional standards.

[0005] In a first aspect, the present invention provides a medical question-and-answer risk control method, comprising: Conduct legitimacy checks on medical questions entered by users, including sensitive word detection. When the legitimacy test result is passed, the question-answering model is used to generate the current answer content corresponding to the medical question content, and the current answer content is streamed out; Based on the continuously updated medical risk control knowledge base, the preset audit model is used to conduct medical risk control testing on the current answer content in real time to obtain the current risk control test results; Based on the current risk control detection results, filter and output the current answer content.

[0006] In an optional embodiment, the legitimacy of the medical question input by the user is checked, including: Based on a pre-built set of rules, sensitive word detection and malicious attack detection are performed on the content of medical questions to obtain legitimacy detection results; Medical Q&A risk control methods also include: When the validity check result is failed, the number of failures within the preset time range is counted; When the number of failures is less than a preset first threshold, a re-entry prompt is given.

[0007] In an optional embodiment, the question-answering model is used to generate the current answer content corresponding to the medical question content, including: Using a pre-built first agent to call a pre-built drug knowledge base, perform preliminary response processing on the medical question content to obtain initial response data; The question-answering model is used in combination with the initial response data to perform secondary response processing on the medical question content to obtain the current answer content.

[0008] In an optional embodiment, based on the continuously iteratively updated medical risk control knowledge base, a preset audit model is used to perform medical risk control testing on the current answer content in real time to obtain the current risk control test results, including: Segment the real-time generated stream of current answers according to the set punctuation marks, and perform sensitive word detection on each paragraph to obtain a first output detection result; Using a pre-built second agent to call a pre-built drug knowledge base, the current answer is tested to see if it violates the drug instructions, and a second output test result is obtained; The audit model is used to call the risk control blacklist and risk control whitelist in the medical risk control knowledge base to perform compliance testing on the current answer content, and the current answer content is risk graded based on the first output test result and the second output test result to obtain the current risk control test result. The current risk control test result includes the illegal content in the current answer content and the target risk level.

[0009] In an optional embodiment, filtering and outputting the current answer content according to the current risk control detection result includes: Determine whether the target risk level in the current risk control detection results reaches the preset risk level; If the preset risk level is reached, the current answer content hit rule is determined and the hit count corresponding to the medical question content is updated; Determining whether the number of hits corresponding to the medical question content reaches a preset second number threshold; If the second numerical threshold is not reached, the output of the current answer content is interrupted, and risk warning words are generated based on the illegal content in the current risk control detection results. The question-answering big model is used to regenerate the current answer content corresponding to the medical question content based on the risk warning words.

[0010] In an optional embodiment, the medical question-and-answer risk control method further includes: If the second number threshold is reached, the output of the current answer content is terminated and a warning of illegal output interception is issued.

[0011] In an optional embodiment, the medical question-and-answer risk control method further includes: If the preset risk level is not reached, continue to output the current answer content, and store the question and answer detection record in the medical risk control knowledge base so that medical personnel can update the risk control blacklist and risk control whitelist in the medical risk control knowledge base.

[0012] In a second aspect, the present invention provides a medical question-and-answer risk control device, comprising: The legality detection module is used to perform legality detection on the medical question content entered by the user, including sensitive word detection; Generate an output module, which is used to generate the current answer content corresponding to the medical question content using the question-answering model when the legitimacy test result is passed, and stream output the current answer content; The risk control detection module is used to perform medical risk control detection on the current answer content in real time using a preset review model based on the continuously iteratively updated medical risk control knowledge base to obtain the current risk control detection results; The filtering output module is used to filter and output the current answer content based on the current risk control detection results.

[0013] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the medical question-and-answer risk control method of any one of the aforementioned embodiments is implemented.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is run by a processor, the medical question-and-answer risk control method of any one of the aforementioned embodiments is executed.

[0015] The medical question-and-answer risk control method and device provided by the present invention can perform a legitimacy check on the medical question content input by the user, and the legitimacy check includes sensitive word detection; when the legitimacy check result is passed, the question-and-answer large model is used to generate the current answer content corresponding to the medical question content, and the current answer content is streamed output; based on the continuously iteratively updated medical risk control knowledge base, the current answer content is subjected to a medical risk control check in real time using a preset audit model to obtain the current risk control check result; based on the current risk control check result, the current answer content is filtered and output. In this way, through the sensitive word detection of the input content, the medical risk control detection of the output content based on the medical risk control knowledge base, the filtered output of the answer content, combined with the continuous iterative update of the medical risk control knowledge base, it is possible to effectively reduce the output of erroneous medical information caused by defects in the AI ​​model, ensure the accuracy, safety and reliability of medical consultations and suggestions, thereby solving the content security risk problem of AI medical products, realizing the construction of a medical risk control system, ensuring that the medical content generated by AI complies with laws, regulations and medical professional standards, and reducing the potential risks of erroneous medical information to the health of the diagnosis and treatment subjects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is an example of DeepSeek's response to breastfeeding medication recommendations; Figure 2 This is the content of some ibuprofen drug instructions; Figure 3 A schematic diagram of a process flow of a medical question-and-answer risk control method provided by an embodiment of the present invention; Figure 4 An example of a re-entry prompt provided by an embodiment of the present invention; Figure 5 A code example for segmented sensitive word detection provided by an embodiment of the present invention; Figure 6 A risk classification diagram of an audit model provided by an embodiment of the present invention; Figure 7 A prompt word segment of an audit model provided by an embodiment of the present invention; Figure 8 This is a diagram of the overall architecture of risk control detection provided by an embodiment of the present invention; Figure 9An overall flow chart of risk control detection provided by an embodiment of the present invention; Figure 10 A schematic diagram of the structure of a medical question-and-answer risk control device provided by an embodiment of the present invention; Figure 11 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] Existing AI medical products pose content security risks because, due to the timeliness of model knowledge and the uncertainty of generated content, their output information may contain a certain degree of accuracy deviation. Given the unique characteristics of the medical industry, which places strict demands on content security, any inaccurate or inappropriate medical advice could pose a significant risk to the patient's health.

[0020] For example, in response to the question “Can I take ibuprofen for pain relief during breastfeeding?”, DeepSeek responded: Figure 1 As shown in the literature, ibuprofen is generally safe for use during breastfeeding. However, some ibuprofen product instructions clearly state that breastfeeding women are prohibited from using it. Figure 2 This example shows the importance and necessity of medical risk control.

[0021] Based on this, the embodiments of the present invention provide a medical question-and-answer risk control method and device, which aims to solve the problem of erroneous content output caused by knowledge lag and response uncertainty in the application of AI models in the medical field, ensure the accuracy, safety and reliability of medical consultation and advice, and thus reduce the potential risks of erroneous medical information to the health of the diagnosis and treatment subjects.

[0022] The embodiments of the present invention mainly solve the following three problems: 1. Model knowledge lag: Medical knowledge is rapidly evolving, with new research findings, diagnostic and treatment guidelines, and clinical experience constantly emerging. However, traditional AI models are typically built based on static training data, making it difficult to timely capture and integrate the latest medical advances. Consequently, their output may lag behind current medical knowledge, and may even produce erroneous or outdated medical recommendations, impacting the accuracy of clinical decision-making.

[0023] 2. Uncertainty in generated responses: Current AI systems based on generative models are subject to a degree of randomness and uncontrollability, and their output may contain inconsistencies, logical flaws, or semantic deviations. This uncertainty is particularly sensitive in medical settings, potentially leading to inaccurate and even misleading medical advice, negatively impacting the patient's health decisions.

[0024] 3. Security risks of medical content: Given the high-risk nature of the medical industry, any erroneous, ambiguous, or inappropriate information may have serious consequences for the life and health of the patient being treated.

[0025] To this end, the embodiments of the present invention propose a dynamic and continuous iteration mechanism for the knowledge base, combined with an optimization strategy for the model output mechanism, to effectively reduce the probability of generating erroneous information, significantly improve the reliability, security, and credibility of the medical AI system, and ensure its compliance and stability in practical applications. Through the technical solutions of the embodiments of the present invention, it is possible to effectively reduce the output of erroneous medical information due to defects in the AI ​​model, providing more accurate and reliable auxiliary support for diagnosis and treatment subjects and medical practitioners.

[0026] The main solutions of the embodiments of the present invention include: 1. Real-time interception of sensitive words: You can access the sensitive word library and intercept input and output content in real time. You can monitor the interception of sensitive words, conduct regular analysis, and maintain whitelists and blacklists of sensitive words.

[0027] 2. Medical risk control for model output content: Leveraging the large model, we employ tiered risk control, such as categorizing risk-free, low-risk, medium-risk, and high-risk. After model output, we conduct real-time risk control, recalling high-risk content and adding risky content to output prompts, prompting the large model to re-output.

[0028] 3. Real-time feedback learning mechanism: When users question AI answers (i.e., output content) or content that poses a risk to risk control, medical personnel will manually correct the results and automatically feed them back to the risk control system to establish a medical risk control knowledge base. A whitelist knowledge base (i.e., a risk control whitelist) will be established for erroneous assessments, and a blacklist knowledge base (i.e., a risk control blacklist) will be established for risk control content.

[0029] To facilitate understanding of this embodiment, a medical question-and-answer risk control method disclosed in an embodiment of the present invention is first introduced in detail.

[0030] The embodiment of the present invention provides a medical question-answering risk control method, which can be executed by an electronic device with data processing capabilities. Figure 3 The flowchart of a medical question-and-answer risk control method shown in FIG. 1 mainly includes the following steps S310 to S330: Step S310: Perform a legitimacy check on the medical question content input by the user, including sensitive word detection.

[0031] To protect user privacy, prevent the spread of harmful information, and ensure that the content complies with relevant laws and regulations, the medical question content entered by the user can be checked for sensitive words. Furthermore, to ensure system security and maintain data integrity and confidentiality, the medical question content entered by the user can also be checked for malicious attacks. Based on this, the above-mentioned legitimacy detection includes sensitive word detection and malicious attack detection, and legitimacy detection can be performed in a rule-based manner. Step S310 above can include: based on a pre-established set of rules, performing sensitive word detection and malicious attack detection on the medical question content to obtain a legitimacy detection result.

[0032] The above rule sets can be constructed based on laws and regulations, business needs, and scenario cases, which may include special characters, sensitive words, malicious instructions, etc., to detect and filter the corresponding content, quickly discover and respond to possible security issues, and ensure data security.

[0033] When the above-mentioned validity check result is passed, step S320 is executed; when the validity check result is failed, the user may be prompted to re-enter.

[0034] In order to ensure the security and stability of the system and protect the security of user data, this embodiment also provides an anti-brute force testing function, which restricts input after multiple hits on sensitive words. Based on this, the above-mentioned medical question-and-answer risk control method also includes: when the legitimacy detection result is a failure, counting the number of failures within a preset time range; when the number of failures is less than the preset first number threshold, prompting re-entry; when the number of failures is greater than or equal to the first number threshold, prohibiting user input.

[0035] The preset time range can be set based on actual needs and is not limited here. For example, the preset time range can be the same day; for another example, the preset time range can be from 24 hours or 12 hours before the current time to the current time. The first number threshold can be set based on actual needs and is not limited here. For example, the first number threshold is 4, which means that if the number of failures exceeds 3, the user is prohibited from entering.

[0036] Take 3 hits on sensitive words, limiting input on the same day as an example. When the number of failures is 2, the re-entry prompt is as follows Figure 4 shown.

[0037] Step S320: When the legitimacy check result is passed, the question-answering model is used to generate the current answer content corresponding to the medical question content, and the current answer content is streamed out.

[0038] To improve the accuracy and reliability of the large question and answer model in processing medical field problems, the large question and answer model can combine the medical field agent to jointly generate output content (i.e. current answer content), wherein the agent can make medical related answers based on the drug knowledge base, and the large question and answer model generates output content combined with the results output by the agent. Based on this, the above step S320 can include: calling the pre-established drug knowledge base by using the pre-constructed first agent to process the medical question content for preliminary reply, and obtaining initial reply data; using the large question and answer model to combine the initial reply data to process the medical question content for secondary reply, and obtaining the current answer content.

[0039] Step S330, according to the continuously updated medical risk control knowledge base, using the preset audit model to detect the current answer content in real time, and obtaining the current risk control detection result.

[0040] The audit model is used to detect whether the output content conforms to the medical related specifications and guidelines. The audit model can be a model selected from a plurality of large models and having the best performance in medical risk control detection.

[0041] In the risk control detection of the output content, not only the audit model can be used, but also sensitive word detection can be performed in real time based on rules, and whether the drug instructions are violated can be detected based on the drug knowledge base, so as to improve the accuracy of the risk control detection result. Based on this, the above step S330 can include: segmenting the real-time generated streaming current answer content according to the set punctuation marks, and performing sensitive word detection on each paragraph to obtain a first output detection result; using the pre-constructed second agent to call the pre-established drug knowledge base to detect whether the current answer content violates the drug instructions to obtain a second output detection result; using the audit model to call the risk control blacklist and risk control whitelist in the medical risk control knowledge base to detect the compliance of the current answer content, and combining the first output detection result and the second output detection result to risk grade the current answer content to obtain the current risk control detection result, which includes the illegal content in the current answer content and the target risk level.

[0042] When performing sensitive word detection on the output content of the large question and answer model, because the output content is streaming, truncation of the output content will affect the accuracy of the sensitive word detection result. The output content can be processed at the code level and segmented and detected according to punctuation marks. An exemplary code snippet is as follows: Figure 5As shown, the steps of sensitive word detection include: using an asyncfor loop to obtain streaming chat content from llm_util.stream_chat_reasoner or llm_util.stream_chat; accumulating the obtained content into current_segment segment by segment; when current_segment contains a specified delimiter (such as a comma, period, etc.), calling the sensitive_word_check function to check for sensitive words in the current segment and returning the test result through yield; finally, if there is any remaining content in current_segment (that is, the last segment), it is also checked for sensitive words and returned.

[0043] This embodiment uses a specific audit model to detect compliance with relevant medical standards and guidelines. A grading system is used to categorize rule-breaking content (i.e., non-compliant content) into high, medium, and low risk levels. In the absence of non-compliant content, the risk is determined to be zero, resulting in a total of four risk levels. It should be noted that the number of risk levels is not limited; alternative embodiments may include three, five, or six risk levels. The medical standards and guidelines utilized by the audit model will also be continuously updated over time to improve its accuracy.

[0044] For example, Figure 6 As shown, after the audit model performs compliance testing on the original output content, it makes risk level judgments, including high risk, medium risk, low risk, and no risk. Among them, high risk requires real-time interception, and medium risk and low risk can be manually reviewed later.

[0045] An example of a prompt word fragment for the audit model is Figure 7 The following is an example of the output of the audit model: [ { "riskLevel": 3, "rule": "This violates the Chinese Pharmacopoeia and drug insert's prohibition on ibuprofen use during breastfeeding. The recommendations for 'short-term use after risk-benefit assessment' and 'wait 24 hours after use to resume breastfeeding' conflict with the prohibition principle." } ] Step S340: filter and output the current answer content according to the current risk control detection result.

[0046] The above-mentioned current risk control detection result may include the target risk level and violation content. Based on this, the above-mentioned step S340 may include: judging whether the target risk level in the current risk control detection result reaches the preset risk level; if it reaches the preset risk level, determining the current answer content hit rule, and updating the number of hits corresponding to the medical question content, that is, the number of hits is increased by 1; judging whether the number of hits corresponding to the medical question content reaches the preset second number threshold; if it does not reach the second number threshold, interrupting the output of the current answer content, generating risk warning words according to the violation content in the current risk control detection result, and using the question and answer big model based on the risk warning words to regenerate the current answer content corresponding to the medical question content.

[0047] The above preset risk level can be set according to actual needs and is not limited here. For example, if the preset risk level is high, when the target risk level is high, the current answer content is determined to have hit the rule. When the target risk level is medium, low, or no risk, the current answer content is determined to have not hit the rule.

[0048] The second threshold can be set based on actual needs and is not limited here. For example, if the second threshold is 5 and the number of hits is less than 5, the Q&A model can be re-output based on the rules violated in the current answer (i.e., illegal content).

[0049] An example of a risk prompt is as follows, where compliance_prompt is the rule that was violated: system_prompt += f""" It was detected that the following information in your last answer does not comply with medical standards and guidelines: {deepseek_chat.compliance_prompt} Please combine the above information and give a standard answer. """ Furthermore, the aforementioned medical Q&A risk control method also includes: if a second threshold is reached, output of the current answer is terminated and a warning message is displayed indicating a violation. If the rule is hit multiple times, output is terminated and the user is prompted, such as by displaying a friendly message explaining why the answer could not be completed and suggesting that they try another query or contact support for assistance. This reduces false positives, improves accuracy, and enhances the user experience.

[0050] Furthermore, the medical Q&A risk control method further includes: if the preset risk level is not reached, continuing to output the current answer content, and storing the Q&A test record in the medical risk control knowledge base, so that medical personnel can update the risk control blacklist and risk control whitelist in the medical risk control knowledge base; wherein the Q&A test record may include the medical question content entered by the user, the answer content, and the risk control test result (the risk control test result may include the violation content and risk level). Exemplarily, when the target risk level is medium, low, or no risk (not reaching the preset risk level), the current answer content may be output in full.

[0051] In this embodiment, a highly reliable and dynamically updated medical risk control knowledge base can be constructed, including: effectively storing each Q&A test record and building it into a risk control knowledge base; medical personnel can regularly annotate the Q&A test records, such as annotating the input medical question content based on the risk level and violation content in the Q&A test records, and constructing and updating risk control blacklists and risk control whitelists to prevent misjudgments and missed judgments; and building a drug knowledge base based on drug instructions to detect whether the output content violates the drug instructions. The drug instructions may include one or more of the following: drug ID, generic name, trade name, specifications, approval number, manufacturer, instructions approval date, instructions revision date, expiration date, warnings (e.g., warning: Do not take if you have a cold or fever), ingredients, and shape.

[0052] The medical question-and-answer risk control method provided by the embodiment of the present invention can effectively reduce the output of erroneous medical information caused by defects in the AI ​​model through sensitive word detection of input content, medical risk control detection of output content based on a medical risk control knowledge base, and filtered output of answer content, combined with continuous iterative updates of the medical risk control knowledge base, thereby ensuring the accuracy, security and reliability of medical consultation and advice, thereby solving the content security risk issues of AI medical products, realizing the construction of a medical risk control system, ensuring that medical content generated by AI complies with laws, regulations and medical professional standards, and reducing the potential risks of erroneous medical information to user health.

[0053] For ease of understanding, refer to Figure 8 The overall architecture of risk control detection is introduced. In this embodiment, the risk control system is mainly divided into three layers, among which the bottom layer is the risk control data center, the middle layer is the risk control strategy center, and the top layer includes input risk control filtering, real-time risk control detection and output risk control filtering.

[0054] Risk Control Data Center: The core infrastructure of the entire risk control system, it undertakes the important responsibilities of data collection, storage, analysis and application, including the medical risk control knowledge base.

[0055] Risk Control Strategy Center: The Risk Control Strategy Center is the core decision-making component of the entire risk control system, responsible for formulating, managing and implementing various risk control rules and strategies.

[0056] Input risk control filtering: Detect sensitive words and malicious attacks on user input content.

[0057] Real-time risk control detection: Based on rules, agent-based review schemes, and audit models, real-time detection of model outputs.

[0058] Output risk control filtering: Filter the risk-controlled content and regenerate the content.

[0059] For ease of understanding, refer to Figure 9 This section describes the overall risk control and detection process.

[0060] like Figure 9 As shown in the figure, the overall risk control detection process includes: After the user enters content, sensitive word detection is performed; Determine whether it contains sensitive words; If yes (that is, it contains sensitive words), prompt the user to re-enter; If not (i.e., it does not contain sensitive words), the large model generates content after processing by the intelligent agent and outputs the content; Conduct medical risk control testing on the content generated by the large model to determine whether it matches the rules; If not (i.e. the rule is not hit), continue outputting the content; If yes (i.e. the rule is hit), determine whether it is a multiple hit; if yes (i.e. multiple hits), terminate content output and prompt the user; if no (i.e. not multiple hits), interrupt content output and risk prompt the large model to re-output.

[0061] In summary, the embodiments of the present invention include the following innovations: 1) Accurately address the core pain points of medical AI: Targeting the two key issues of knowledge lag and response uncertainty, this approach effectively improves the timeliness and accuracy of medical advice by synchronizing with the latest medical advances in real time and optimizing generation mechanisms.

[0062] 2) Strengthening the security of medical content: Through continuous iteration of the knowledge base and model optimization, the risk of outputting erroneous or misleading information is significantly reduced, providing highly reliable medical consultation guarantees for patients.

[0063] 3) Enhance industry credibility: Combining rigorous mechanisms designed for the specificities of the medical field, this significantly enhances the professionalism and security of AI-generated content, and helps normalize and standardize the development of medical AI applications.

[0064] 4) Focusing on the health of the patient: proactively avoiding potential health risks through technical means, demonstrating a deep control over the seriousness of medical information, and providing scientific, safe, and intelligent support for the patient's decision-making.

[0065] Corresponding to the above-mentioned medical question-answering risk control method, an embodiment of the present invention further provides a medical question-answering risk control device. Figure 10 The schematic diagram of a medical question-and-answer risk control device is shown, and the device includes: The legality detection module 1001 is used to perform legality detection on the medical question content input by the user, including sensitive word detection; The generation output module 1002 is used to generate the current answer content corresponding to the medical question content using the question-answering model when the legitimacy test result is passed, and to stream output the current answer content; The risk control detection module 1003 is used to perform medical risk control detection on the current answer content in real time using a preset audit model based on the continuously iteratively updated medical risk control knowledge base to obtain the current risk control detection result; The filtering and outputting module 1004 is used to filter and output the current answer content according to the current risk control detection result.

[0066] The medical question-and-answer risk control device provided by the embodiment of the present invention can effectively reduce the output of erroneous medical information due to defects in the AI ​​model through sensitive word detection of input content, medical risk control detection of output content based on a medical risk control knowledge base, and filtered output of answer content, combined with continuous iterative updates of the medical risk control knowledge base, thereby ensuring the accuracy, safety and reliability of medical consultation and advice, thereby solving the content security risk issues of AI medical products, realizing the construction of a medical risk control system, ensuring that medical content generated by AI complies with laws, regulations and medical professional standards, and reducing the potential risks of erroneous medical information to user health.

[0067] Furthermore, the legality detection module 1001 is specifically used to: perform sensitive word detection and malicious attack detection on the content of the medical question based on a pre-built rule set to obtain a legality detection result; The above device also includes an illegal processing module, which is used to count the number of failures within a preset time range when the legitimacy detection result is a failure; when the number of failures is less than a preset first threshold, a re-entry prompt is given.

[0068] Furthermore, the above-mentioned generation output module 1002 is specifically used to: use the pre-built first intelligent agent to call the pre-established drug knowledge base to perform preliminary response processing on the content of the medical question to obtain initial response data; use the question-answering big model combined with the initial response data to perform secondary response processing on the content of the medical question to obtain the current answer content.

[0069] Furthermore, the above-mentioned risk control detection module 1003 is specifically used to: segment the real-time generated streaming current answer content according to the set punctuation marks, and perform sensitive word detection on each paragraph to obtain a first output detection result; use a pre-built second intelligent agent to call a pre-established drug knowledge base to detect whether the current answer content violates the drug instructions, and obtain a second output detection result; use the audit model to call the risk control blacklist and risk control whitelist in the medical risk control knowledge base to perform compliance detection on the current answer content, and combine the first output detection result and the second output detection result to perform risk classification on the current answer content to obtain the current risk control detection result, which includes the illegal content and target risk level in the current answer content.

[0070] Furthermore, the above-mentioned filtering output module 1004 is specifically used to: determine whether the target risk level in the current risk control detection result reaches the preset risk level; if it reaches the preset risk level, determine the current answer content hit rule, and update the number of hits corresponding to the medical question content; determine whether the number of hits corresponding to the medical question content reaches the preset second number threshold; if it does not reach the second number threshold, interrupt the output of the current answer content, generate risk warning words according to the illegal content in the current risk control detection result, and use the question and answer big model based on the risk warning words to regenerate the current answer content corresponding to the medical question content.

[0071] Furthermore, the filtering output module 1004 is also used to: if the second number threshold is reached, terminate the output of the current answer content and issue a warning of illegal output interception.

[0072] Furthermore, the above-mentioned filtering output module 1004 is also used to: if the preset risk level is not reached, continue to output the current answer content, and store the question and answer detection record in the medical risk control knowledge base, so that medical personnel can update the risk control blacklist and risk control whitelist in the medical risk control knowledge base.

[0073] The medical question-and-answer risk control device provided in this embodiment has the same implementation principle and technical effects as those of the aforementioned medical question-and-answer risk control method embodiment. For the sake of brief description, for matters not mentioned in the medical question-and-answer risk control device embodiment, reference may be made to the corresponding content in the aforementioned medical question-and-answer risk control method embodiment.

[0074] like Figure 11 As shown, an embodiment of the present invention provides an electronic device 1100, including: a processor 1101, a memory 1102 and a bus. The memory 1102 stores a computer program that can be run on the processor 1101. When the electronic device 400 is running, the processor 1101 and the memory 1102 communicate through the bus, and the processor 1101 executes the computer program to implement the above-mentioned medical question and answer risk control method.

[0075] Specifically, the memory 1102 and processor 1101 can be general-purpose memories and processors, which are not specifically limited here.

[0076] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, executes the medical question-and-answer risk control method described in the preceding method embodiments. Such computer-readable storage media include various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), RAM, a magnetic disk, or an optical disk.

[0077] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0078] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not limiting, and thus other examples of the exemplary embodiments may have different values.

[0079] The flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, the indirect coupling or communication connection of the device or module can be electrical, mechanical or other forms.

[0081] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0082] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical question-answering risk control method, characterized in that: include: Performing a legitimacy check on the medical question content input by the user, including sensitive word detection; When the legitimacy test result is passed, the question-answering model is used to generate the current answer content corresponding to the medical question content, and the current answer content is streamed output; Based on the continuously iteratively updated medical risk control knowledge base, the preset audit model is used to perform medical risk control testing on the current answer content in real time to obtain the current risk control test results; According to the current risk control detection result, the current answer content is filtered and output.

2. The medical question-and-answer risk control method according to claim 1, characterized in that: The legitimacy check of the medical question content input by the user includes: Based on a pre-built set of rules, the medical question content is tested for sensitive words and malicious attacks to obtain a legitimacy test result; The medical question-answering risk control method further includes: When the validity check result is failure, count the number of failures within a preset time range; When the number of failures is less than a preset first threshold, a re-entry prompt is given.

3. The medical question-answering risk control method according to claim 1, characterized in that: The method of using the question-answering model to generate the current answer content corresponding to the medical question content includes: Using a pre-built first agent to call a pre-built drug knowledge base, perform preliminary answer processing on the medical question content to obtain initial answer data; The question-answering model is used in combination with the initial answer data to perform secondary answer processing on the medical question content to obtain the current answer content.

4. The medical question-answering risk control method according to claim 1, characterized in that: The medical risk control knowledge base that is continuously updated and iterated is used to perform medical risk control testing on the current answer content in real time using a preset audit model to obtain the current risk control testing results, including: Segmenting the real-time generated stream of the current answer content according to set punctuation marks, and performing sensitive word detection on each paragraph to obtain a first output detection result; Using a pre-built second agent to call a pre-built drug knowledge base, the current answer is tested to see if it violates the drug instructions, and a second output test result is obtained; The audit model is used to call the risk control blacklist and risk control whitelist in the medical risk control knowledge base to perform compliance detection on the current answer content, and the current answer content is risk graded in combination with the first output detection result and the second output detection result to obtain the current risk control detection result, which includes the illegal content and target risk level in the current answer content.

5. The medical question-and-answer risk control method according to claim 1, characterized in that: The filtering and outputting of the current answer content according to the current risk control detection result includes: Determine whether the target risk level in the current risk control detection result reaches the preset risk level; If the preset risk level is reached, determining the hit rule of the current answer content and updating the hit count corresponding to the medical question content; Determining whether the number of hits corresponding to the medical question content reaches a preset second number threshold; If the second number threshold is not reached, the output of the current answer content is interrupted, a risk warning word is generated according to the illegal content in the current risk control detection result, and the current answer content corresponding to the medical question content is regenerated based on the risk warning word using the question-answer model.

6. The medical question-and-answer risk control method according to claim 5, characterized in that: The medical question-answering risk control method further includes: If the second number threshold is reached, the output of the current answer content is terminated, and a warning of illegal output interception is issued.

7. The medical question-and-answer risk control method according to claim 5, characterized in that: The medical question-answering risk control method further includes: If the preset risk level is not reached, continue to output the current answer content, and store the question and answer detection record in the medical risk control knowledge base for medical personnel to update the risk control blacklist and risk control whitelist in the medical risk control knowledge base.

8. A medical question-and-answer risk control device, characterized in that: include: A legality detection module, configured to perform legality detection on the medical question content input by the user, wherein the legality detection includes sensitive word detection; A generation output module is used to generate a current answer content corresponding to the medical question content using the question-answering model when the legitimacy test result is passed, and to perform streaming output of the current answer content; A risk control detection module is used to perform medical risk control detection on the current answer content in real time using a preset audit model based on a continuously iteratively updated medical risk control knowledge base to obtain the current risk control detection result; The filtering output module is used to filter and output the current answer content according to the current risk control detection result.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the medical question-and-answer risk control method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the medical question-and-answer risk control method according to any one of claims 1 to 7 is executed.

Images