Female pelvic tumor knowledge dialogue question-answering method and system based on large model

Abstract

The invention discloses a female pelvic tumor knowledge dialogue questioning and answering method and system based on a large model, and the method comprises the steps: S1, carrying out the preprocessing of a question text inputted by a user, extracting the text features, carrying out the mapping of an identified named entity, and carrying out the recognition of the named entity; performing female pelvic tumor knowledge retrieval on the mapped named entity by using a knowledge graph to obtain a female pelvic tumor knowledge retrieval result; s2, performing structured processing on existing case data uploaded by a user by using OCR (Optical Character Recognition), and performing prescription character feature extraction on key information identified by processing by using a large model; s3, constructing a female pelvic tumor large model, and extracting and inputting the female pelvic tumor knowledge retrieval result and the prescription character features into the female pelvic tumor large model for training to obtain a trained female pelvic tumor large model; and S4, inputting real-time multi-modal data into the trained female pelvic tumor large model to obtain a female pelvic tumor knowledge dialogue question and answer result.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and medical application technology, and in particular to a female pelvic tumor knowledge dialogue question-answering method and system based on a large model. Background Art

[0002] Pelvic tumor is one of the most common malignant tumors in gynecology. At present, pelvic tumor disease has been defined as a chronic disease by the World Health Organization, and the diagnosis and treatment of pelvic diseases is a long process. There are no obvious specific symptoms in the early stage of the disease, and most patients are in the late stage when they seek medical treatment. Its treatment and management have the characteristics of long cycle, high cost, difficult supervision, old age of patients, and high mortality rate, which require early detection, treatment and control. However, since there are no obvious specific symptoms in the early stage of the disease, most patients are in the late stage when they seek medical treatment. Since most female patients with pelvic tumors lack the corresponding recuperation knowledge, they need to seek medical resources, but medical resources are limited. During the long diagnosis and treatment process, although the medical needs of some users or patients are relatively simple, they all need to go through the offline medical process, which makes the contradiction between medical resources and medical needs more prominent.

[0003] The emergence of large models (such as GPT-4, Wenxin Yiyan, Tongyi Qianwen, Baichuan3, RadFM multimodal model, etc.) has promoted the rapid development of artificial intelligence large model technology. Based on a large amount of network data corpus and powerful architecture, large models can already read, write, listen and talk like humans. Their multimodal learning ability can integrate various types of medical data such as text, images, and sounds to provide more comprehensive and accurate analysis and diagnosis. Secondly, the deep reasoning ability of large models helps to make complex medical decisions. In addition, stable performance and knowledge update capabilities ensure the reliability and timeliness of medical advice. At the same time, the language understanding and generation capabilities of large models enable them to handle professional terms and complex sentences. Therefore, the use of multimodal large models for tumor auxiliary diagnosis and treatment has become a realistic and feasible solution. Large models refer to machine learning or deep learning models with huge parameter quantities and computing resource requirements. These models are usually composed of billions to tens of billions or even more trainable parameters, far exceeding traditional models. The emergence of large models is due to two main factors. One is the growth of data scale. With the popularization of the Internet and the improvement of data collection capabilities, a large amount of training data has become available. More data can help large models learn more complex and accurate feature representations, thereby improving their performance. Second, the development of computing resources. With the continuous advancement of hardware technology, such as graphics processing units (GPUs), tensor processing units (TPUs), and distributed computing, the training and reasoning of large models have become more feasible. These technologies provide powerful computing power, allowing large models to be trained and deployed in a reasonable time. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a female pelvic tumor knowledge dialogue question-answering method and system based on a large model, the method comprising:

[0005] Step S1, after preprocessing the question text input by the user and extracting text features, the identified named entities are mapped, and the mapped named entities are used to perform female pelvic tumor knowledge retrieval results using the knowledge graph;

[0006] Step S2: Use OCR to perform structured processing on the existing case data uploaded by the user, and use a large model to extract prescription text features from the key information identified by the processing;

[0007] Step S3, constructing a large model of female pelvic tumors, and inputting the female pelvic tumor knowledge retrieval results and the prescription text feature extraction into the large model of female pelvic tumors for training, to obtain a trained large model of female pelvic tumors;

[0008] Step S4: input the real-time multimodal data into the trained large model of female pelvic tumors to obtain the results of the female pelvic tumor knowledge dialogue question and answer.

[0009] Optionally, in step S1, after preprocessing the question text input by the user and extracting text features, the identified named entities are mapped, and the mapped named entities are used to perform female pelvic tumor knowledge retrieval results using the knowledge graph, and the content specifically includes:

[0010] Use the bert-base-chinese model to extract features from the preprocessed question text to obtain question text features;

[0011] Obtaining named entities based on the question text features, mapping the named entities, and converting the question text input by the user into machine language;

[0012] The machine language is input into the knowledge graph for retrieval to obtain female pelvic tumor knowledge retrieval results.

[0013] Optionally, the construction process of the knowledge graph specifically includes:

[0014] Preprocess the existing medical knowledge on female pelvic tumors to obtain preprocessed data;

[0015] Using natural language processing to perform polarity analysis on the preprocessed data and extract entity, relationship and attribute information;

[0016] The entities, relationships and attribute information are organized into a graph structure, and a knowledge graph is constructed based on the graph structure.

[0017] Optionally, in step S2, the process of using OCR to perform structured processing on the existing case data uploaded by the user and extracting prescription text features using a large model using the key information identified by the processing specifically includes:

[0018] Use OCR to structure the existing case data uploaded by users to obtain structured data;

[0019] Segmenting the structured data to obtain a number of word texts, and performing spelling check and correction on the word texts;

[0020] Use a binary decision tree to clean and remove duplicates from the checked and corrected word text to obtain key information;

[0021] The key information is used to extract prescription text features using a large model.

[0022] The present invention also discloses a female pelvic tumor knowledge dialogue question-answering system based on a large model, the system comprising:

[0023] The knowledge retrieval module is used to pre-process the question text input by the user and extract text features, then map the identified named entities, and use the knowledge graph to retrieve the knowledge results of female pelvic tumors based on the mapped named entities;

[0024] The prescription text extraction module is used to use OCR to perform structured processing on the existing case data uploaded by the user, and use the large model to extract prescription text features from the key information identified;

[0025] A large model question-answering module is used to construct a large model of female pelvic tumors, based on the female pelvic tumor knowledge retrieval results and the prescription text feature extraction input into the female pelvic tumor large model for training, to obtain a trained large model of female pelvic tumors;

[0026] The virtual physician dialogue module is used to input real-time multimodal data into the trained large model of female pelvic tumors to obtain female pelvic tumor knowledge dialogue question and answer results.

[0027] Optionally, the workflow of the knowledge retrieval module specifically includes:

[0028] Use the bert-base-chinese model to extract features from the preprocessed question text to obtain question text features;

[0029] Obtaining named entities based on the question text features, mapping the named entities, and converting the question text input by the user into machine language;

[0030] The machine language is input into the knowledge graph for retrieval to obtain female pelvic tumor knowledge retrieval results.

[0031] Optionally, the construction process of the knowledge graph specifically includes:

[0032] Preprocess the existing medical knowledge on female pelvic tumors to obtain preprocessed data;

[0033] Using natural language processing to perform polarity analysis on the preprocessed data and extract entity, relationship and attribute information;

[0034] The entities, relationships and attribute information are organized into a graph structure, and a knowledge graph is constructed based on the graph structure.

[0035] Optionally, the workflow of the prescription text extraction module specifically includes:

[0036] Use OCR to structure the existing case data uploaded by users to obtain structured data;

[0037] Segmenting the structured data to obtain a number of word texts, and performing spelling check and correction on the word texts;

[0038] Use a binary decision tree to clean and remove duplicates from the checked and corrected word text to obtain key information;

[0039] The key information is used to extract prescription text features using a large model.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The present invention can effectively process complex features and user behavior patterns by fine-tuning large language models (such as GLM-4-9B), solves the problem that traditional models are insufficient in processing discrete variables, and improves the accuracy of data analysis; 2. The present invention combines natural language processing with knowledge graph technology to help the system understand the problem background more comprehensively and deeply, provide more targeted answers, improve the efficiency and accuracy of data query, and improve user experience; 3. Dynamically updating the knowledge graph can not only obtain the latest medical information in a timely manner, but also continuously improve its own knowledge base based on user feedback to ensure that the information provided is accurate and cutting-edge; 4. The present invention connects large language models with graph databases (such as Neo4j) through Langchain technology, which improves data processing speed, flexibility and scalability, and is suitable for processing large-scale data sets; 5. The present invention uses the video form of virtual doctors to allow users to experience more vivid and concrete feedback. Compared with traditional text replies, video replies can increase user trust and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 A method step diagram of a female pelvic tumor knowledge dialogue question-answering method based on a large model according to an embodiment of the present invention;

[0044] Figure 2 This is a use case diagram of a method and system for virtual physician dialogue with a large female pelvic tumor model according to an embodiment of the present invention;

[0045] Figure 3 This is a knowledge retrieval sequence diagram of an embodiment of the female pelvic tumor large model virtual physician dialogue method and system of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] Embodiment 1

[0049] A female pelvic tumor knowledge dialogue question-answering method based on a large model, such as Figure 1 As shown, the method includes:

[0050] Step S1: After preprocessing the question text input by the user and extracting text features, the identified named entities are mapped, and the mapped named entities are used to retrieve the knowledge results of female pelvic tumors. The user input text is preprocessed by word segmentation, part-of-speech tagging and other preprocessing operations; the BERT-base-Chinese model is used to extract text features from documents such as medical patents, medical literature, and medical records. The pre-trained BERT-base-Chinese model can effectively identify the named entities therein; the identified named entities are mapped according to the predefined word slot types, and the user's natural language questions are converted into query languages ​​that can operate on the knowledge graph; the GraphQAChain in LangChain is responsible for taking over the query language that can operate on the knowledge graph and submitting it to the medical knowledge graph of female pelvic tumors to obtain accurate female pelvic tumor knowledge retrieval answers.

[0051] In this embodiment, predefined word slot types are labels for understanding and processing different semantic elements in the user's natural language input. These word slot types are generally used to extract key information from the text and convert natural language questions into structured queries. The following are the predefined word slot types and their parsing that may be involved in the text: disease names (for example: pelvic tumors, uterine cancer, ovarian cancer, etc.), which describe the types of tumors that may occur in the female pelvis, and clarify the specific disease that the user is asking about; symptoms (for example: abdominal pain, bleeding, difficulty urinating, etc.), which describe the symptoms or manifestations that may be associated with female pelvic tumors, and are used to identify issues that the user is concerned about; treatment methods (for example: surgery, chemotherapy, radiotherapy, traditional Chinese medicine treatment, etc.), which indicate that the user is interested in the treatment plan for the disease, and are used to retrieve relevant treatment information; examination methods (for example: ultrasound examination, CT scan, MRI, etc.), which are used to ask questions and specific diagnostic examinations Questions related to the means; pathological type (for example, serous tumor, mucinous tumor, malignant cyst, etc.), which identifies the pathological type information of the disease and helps the system to further clarify the specific tumor type; drug name (for example, paclitaxel, cisplatin, leuprorelin acetate, etc.), which describes the drugs used for specific tumors and is used to retrieve medication plans or side effects; patient characteristics (for example, age, medical history, family history, etc.), which are very important for the selection of individualized treatment plans and consideration of indications; site of onset (for example, pelvic cavity, uterus, ovary, etc.), which specifically describes the site of occurrence of the tumor or disease and helps to query the treatment or symptom information of a specific site.

[0052] The process of constructing a knowledge graph includes: obtaining a data set, performing text recognition on the data set to obtain a text data set; extracting entity relationships from the text data set; constructing a graph model based on the extracted data, and storing the constructed graph model in a knowledge graph database.

[0053] Specifically, we obtain data sets, such as public data sets, Web crawling data, business data, etc. The data forms include structured data, semi-structured data, and unstructured data; then, we preprocess the obtained data sets, including removing noise, correcting errors, standardizing data, etc., to ensure data quality, and then use natural language processing technology to analyze text data, including tasks such as word segmentation, part-of-speech tagging, and entity recognition to extract useful entity, relationship, and attribute information. Finally, we organize the entity, relationship, and attribute information in the text data into a graph structure to build a knowledge graph database. The knowledge graph is used to represent the structure and relationship between data; GraphQAChain parses user queries and maps the query results to the relevant entities and relationships in the graph database; machine learning algorithms are used to extract relevant information from the knowledge graph, and the corresponding answers are generated through the female pelvic tumor large model question and answer module.

[0054] Step S2: Use OCR to perform structured processing on the existing case data uploaded by the user, and use a large model to extract prescription text features from the key information identified during processing. Obtain the medical record images, medical record documents, doctor-patient conversation data, etc. uploaded by the user, determine the type of data uploaded by the user, if it is a text format file, directly obtain the plain text information; if it is a PDF or image format file, use OCR technology to identify and obtain the text data in the document file, perform structured processing on the medical record image input by the user, obtain the structured text of the medical record, and identify the patient's disease, symptoms, signs, medical history, allergy history and other key information; if it is doctor-patient conversation data, transcribe it into text data, and extract the patient's disease, symptoms, signs, medical history and allergy history and other key information from the conversation; use the bert-base-chinese model to extract text features in the medical record text data, effectively identify the named entities therein, map the identified named entities according to the predefined word slot type, and convert the user's natural language questions into a query language that can operate on the knowledge graph; through the GraphQAChain in LangChain, it is responsible for taking over the query language that can operate on the knowledge graph and submitting it to the knowledge graph to obtain accurate recommended drugs.

[0055] When it is determined that the type of data uploaded by the user is a medical record image, the Transformer-based OCR recognition technology is used to perform text recognition and text extraction on the medical record image; the recognized text is segmented to obtain several sentence texts, and professional medical vocabulary, common abbreviations and term libraries are collected to build a medical dictionary. The edit distance algorithm (such as Levenshtein distance) is used to detect spelling errors, and the most similar medical vocabulary is matched for correction. Finally, redundant sentence texts in the text to be cleaned are eliminated based on a binary decision tree; the useful text data of the medical record is structured to obtain the structured text of the medical record.

[0056] When it is determined that the type of data uploaded by the user is doctor-patient conversation data, the audio file is converted into audio data in a standard format in real time, and the converted audio data is compressed. A high-performance speech recognition service is used to transcribe the audio data into text data and synchronize it with the original audio data to obtain text data. Non-language characters and noise are removed from the obtained text data. A differentiated text cleaning strategy is implemented based on the differences between the natural language spoken by the patient and the professional terminology of the doctor. The doctor-patient roles and contextual information are combined to judge and correct the ambiguity and errors of speech recognition based on the contextual information. The professional terminology used by the doctor and the daily language used by the patient are compared. Standardized dictionaries are established for different languages ​​and differentiated processing is performed. For special vocabulary in the medical field and patients' spoken expressions, word segmentation and part-of-speech tagging algorithms are optimized, and domain-specific vocabulary libraries and grammatical rules are introduced to ensure the correct segmentation and part-of-speech tagging of medical terms. For entity and relationship types in the medical field, special named entity recognition and relationship extraction models are established to distinguish between symptom entities described by patients and diagnosis entities described by doctors based on the principle of separation of doctor-patient roles. On the basis of text structuring, the languages ​​of doctors and patients are processed separately according to the characteristics of doctor-patient dialogues. All dialogue text data are structured to obtain structured text of medical records.

[0057] Step S3, construct a large model of female pelvic tumors, and input the female pelvic tumor knowledge retrieval results and the prescription text feature extraction into the large model of female pelvic tumors for training to obtain a trained large model of female pelvic tumors. Train and construct a large model of female pelvic tumors, and use the large model of female pelvic tumors to respond to the input multimodal data through question and answer.

[0058] The construction process of the large model of female pelvic tumors specifically includes: selecting the original model; selecting the fine-tuning dataset, the ChatMed dataset, and the Chinese Medical Dialogue dataset; the queries (or prompts) in the ChatMed dataset and the Chinese Medical Dialogue dataset come from medical consultation questions on the Internet (110, 113), covering multiple topics such as disease diagnosis and medication guidance, reflecting the medical consultation needs of different users / patients in the real world. The datasets are all Chinese datasets, targeted at Chinese users, and more suitable for Chinese questions and answers. Collect and construct a variety of datasets related to female pelvic tumors, including medical records, consultation dialogues, and medical literature; use the fine-tuning dataset to fine-tune the model so that it has the ability to handle issues related to the diagnosis and treatment of female pelvic tumors, forming a more targeted fine-tuning model.

[0059] In this embodiment, GLM-4-9B is selected as the original model.

[0060] Acquire historical multimodal data uploaded by users, and conduct multiple rounds of question and answering through the female pelvic tumor large model question and answer module; multimodal data includes report data, medical record report data, medical record images, and consultation voice data. When users upload report data, all information is used as input in combination with the report data uploaded by doctors, the user's medical record report data, and prompt templates for specific diagnosis and treatment results and treatment plan reports; when users upload real-time consultation voice data, the conversation is recorded and collected in real time, and transcribed into text data, and all information is used as input; the female pelvic tumor large model is used to perform inference question and answer responses to generate diagnosis and treatment results and treatment plan reports.

[0061] After each round of conversation, the system uses natural language processing technology to extract key medical entities and relationships from the conversation content; the extracted information is compared with the existing knowledge in the knowledge graph. If new information is found, the difference is detected through the comparison algorithm, and the knowledge fusion and update mechanism is used to integrate the new information into the knowledge graph in real time.

[0062] Step S4: input the real-time multimodal data into the trained large model of female pelvic tumors to obtain the results of the female pelvic tumor knowledge dialogue question and answer.

[0063] Receive the response result returned by the large model of female pelvic tumor, and generate video generation parameters in combination with the user ID and the virtual physician ID; create the virtual physician's response video through the video generation platform, and then parse the video address and play it to the user.

[0064] The user can select a virtual doctor image, and each virtual doctor image will correspond to a virtual doctor ID; the virtual doctor video generation parameters are formed according to the reply content of the interactive conversation, the requesting user identifier and the virtual doctor ID, and the virtual doctor video generation parameters and the virtual doctor video generation request are sent to the virtual human video generation platform; the virtual human video generation platform generates relevant reply videos according to the virtual doctor video generation parameters, and returns the reply video address by calling the video generation interface; the reply video address is parsed according to the native video tag to obtain the reply video, and the virtual doctor reply video is played to the user.

[0065] In this embodiment, the knowledge retrieval process of the female pelvic tumor knowledge dialogue question-answering system is as follows: Figure 3As shown, specifically: the user sends the conversation content to the female pelvic tumor large model virtual doctor question-and-answer system. After receiving the user input, the system starts to process the request. The system requests to pass data to the knowledge retrieval module, which is used to preprocess the question text entered by the user and extract text features, map the identified named entities, use the knowledge graph to perform female pelvic tumor knowledge retrieval results on the mapped named entities, and convert the user's input into a query that can be retrieved in the knowledge graph. The knowledge retrieval module returns the answer retrieved from the knowledge graph based on the user input content and the corresponding preprocessing operations. This step obtains the specific answer required by the user by matching relevant diseases, symptoms or treatment information in the knowledge graph. The returned answer is passed to the integration module of the user content and the knowledge graph, which is responsible for natural language processing of the original answer so that the answer content is more in line with user needs and natural language expression habits. The integrated answer content is returned to the virtual physician dialogue module, and the system is ready to pass the information to the user. If the user has further needs, the system will initiate a request to call the virtual doctor video module, transfer to the virtual doctor generation module, receive the response result returned by the large model of female pelvic tumors, and generate video generation parameters based on the user ID and virtual physician ID; the virtual doctor generation module generates a virtual doctor's response video according to the request, including the answer to the user's question and the corresponding emotional video performance. The system parses the video address of the virtual doctor and prepares to play it to the user. The system will eventually play the answer content and the virtual doctor video to the user, and the user can get further answers in the form of audio and video, so as to better understand the answer content.

[0066] Embodiment 2

[0067] A female pelvic tumor knowledge dialogue question answering system based on a large model, such as Figure 2 As shown, the system comprises:

[0068] The knowledge retrieval module is used to preprocess the question text input by the user and extract text features, then map the identified named entities, and use the knowledge graph to retrieve the knowledge results of female pelvic tumors.

[0069] Step S1: After preprocessing the question text input by the user and extracting text features, the identified named entities are mapped, and the mapped named entities are used to retrieve the knowledge results of female pelvic tumors. The user input text is preprocessed by word segmentation, part-of-speech tagging and other preprocessing operations; the BERT-base-Chinese model is used to extract text features from documents such as medical patents, medical literature, and medical records. The pre-trained BERT-base-Chinese model can effectively identify the named entities therein; the identified named entities are mapped according to the predefined word slot types, and the user's natural language questions are converted into query languages ​​that can operate on the knowledge graph; the GraphQAChain in LangChain is responsible for taking over the query language that can operate on the knowledge graph and submitting it to the medical knowledge graph of female pelvic tumors to obtain accurate female pelvic tumor knowledge retrieval answers.

[0070] The process of constructing a knowledge graph includes: obtaining a data set, performing text recognition on the data set to obtain a text data set; extracting entity relationships from the text data set; constructing a graph model based on the extracted data, and storing the constructed graph model in a knowledge graph database.

[0071] Specifically, we obtain data sets, such as public data sets, Web crawling data, business data, etc. The data forms include structured data, semi-structured data, and unstructured data; then, we preprocess the obtained data sets, including removing noise, correcting errors, standardizing data, etc., to ensure data quality, and then use natural language processing technology to analyze text data, including tasks such as word segmentation, part-of-speech tagging, and entity recognition to extract useful entity, relationship, and attribute information. Finally, we organize the entity, relationship, and attribute information in the text data into a graph structure to build a knowledge graph database. The knowledge graph is used to represent the structure and relationship between data; GraphQAChain parses user queries and maps the query results to the relevant entities and relationships in the graph database; machine learning algorithms are used to extract relevant information from the knowledge graph, and the corresponding answers are generated through the female pelvic tumor large model question and answer module.

[0072] The prescription text extraction module is used to use OCR to perform structured processing on the existing case data uploaded by users, and use a large model to extract prescription text features from the key information identified.

[0073] Obtain the medical record images, medical record documents, doctor-patient conversation data, etc. uploaded by the user, determine the type of data uploaded by the user, if it is a text format file, directly obtain the plain text information; if it is a PDF or image format file, use OCR technology to identify and obtain the text data in the document file, perform structured processing on the medical record image input by the user, obtain the structured text of the medical record, and identify the patient's disease, symptoms, signs, medical history, allergy history and other key information; if it is doctor-patient conversation data, transcribe it into text data, and extract the patient's disease, symptoms, signs, medical history and allergy history and other key information from the conversation; use the bert-base-chinese model to extract text features in the medical record text data, effectively identify the named entities therein, map the identified named entities according to the predefined word slot type, and convert the user's natural language questions into a query language that can operate on the knowledge graph; through the GraphQAChain in LangChain, it is responsible for taking over the query language that can operate on the knowledge graph and submitting it to the knowledge graph to obtain accurate recommended drugs.

[0074] When it is determined that the type of data uploaded by the user is a medical record image, the Transformer-based OCR recognition technology is used to perform text recognition and text extraction on the medical record image; the recognized text is segmented to obtain several sentence texts, and professional medical vocabulary, common abbreviations and term libraries are collected to build a medical dictionary. The edit distance algorithm (such as Levenshtein distance) is used to detect spelling errors, and the most similar medical vocabulary is matched for correction. Finally, redundant sentence texts in the text to be cleaned are eliminated based on a binary decision tree; the useful text data of the medical record is structured to obtain the structured text of the medical record.

[0075] When it is determined that the type of data uploaded by the user is doctor-patient conversation data, the audio file is converted into audio data in a standard format in real time, and the converted audio data is compressed. A high-performance speech recognition service is used to transcribe the audio data into text data and synchronize it with the original audio data to obtain text data. Non-language characters and noise are removed from the obtained text data. A differentiated text cleaning strategy is implemented based on the differences between the natural language spoken by the patient and the professional terminology of the doctor. The doctor-patient roles and contextual information are combined to judge and correct the ambiguity and errors of speech recognition based on the contextual information. The professional terminology used by the doctor and the daily language used by the patient are compared. Standardized dictionaries are established for different languages ​​and differentiated processing is performed. For special vocabulary in the medical field and patients' spoken expressions, word segmentation and part-of-speech tagging algorithms are optimized, and domain-specific vocabulary libraries and grammatical rules are introduced to ensure the correct segmentation and part-of-speech tagging of medical terms. For entity and relationship types in the medical field, special named entity recognition and relationship extraction models are established to distinguish between symptom entities described by patients and diagnosis entities described by doctors based on the principle of separation of doctor-patient roles. On the basis of text structuring, the languages ​​of doctors and patients are processed separately according to the characteristics of doctor-patient dialogues. All dialogue text data are structured to obtain structured text of medical records.

[0076] The large model question-answering module is used to construct a large model of female pelvic tumors. The female pelvic tumor knowledge retrieval results and the prescription text feature extraction are input into the large model of female pelvic tumors for training to obtain a trained large model of female pelvic tumors.

[0077] The construction process of the large model of female pelvic tumors specifically includes: selecting the original model; selecting the fine-tuning dataset, the ChatMed dataset, and the Chinese Medical Dialogue dataset; the queries (or prompts) in the ChatMed dataset and the Chinese Medical Dialogue dataset come from medical consultation questions on the Internet (110, 113), covering multiple topics such as disease diagnosis and medication guidance, reflecting the medical consultation needs of different users / patients in the real world. The datasets are all Chinese datasets, targeted at Chinese users, and more suitable for Chinese questions and answers. Collect and construct a variety of datasets related to female pelvic tumors, including medical records, consultation dialogues, and medical literature; use the fine-tuning dataset to fine-tune the model so that it has the ability to handle issues related to the diagnosis and treatment of female pelvic tumors, forming a more targeted fine-tuning model.

[0078] In this embodiment, GLM-4-9B is selected as the original model.

[0079] Acquire multimodal data uploaded by users, and conduct multiple rounds of question and answering through the female pelvic tumor large model question and answer module; multimodal data includes report data, medical record report data, medical record images, and consultation voice data. When users upload report data, all information is used as input in combination with the report data uploaded by doctors, the user's medical record report data, and prompt templates for specific diagnosis and treatment results and treatment plan reports; when users upload real-time consultation voice data, the conversation is recorded and collected in real time and transcribed into text data, and all information is used as input; the female pelvic tumor large model is used to perform inference question and answer responses to generate diagnosis and treatment results and treatment plan reports.

[0080] After each round of conversation, the system uses natural language processing technology to extract key medical entities and relationships from the conversation content; the extracted information is compared with the existing knowledge in the knowledge graph. If new information is found, the difference is detected through the comparison algorithm, and the knowledge fusion and update mechanism is used to integrate the new information into the knowledge graph in real time.

[0081] The virtual physician dialogue module is used to input real-time multimodal data into the trained large model of female pelvic tumors to obtain female pelvic tumor knowledge dialogue question and answer results.

[0082] Receive the response result returned by the large model of female pelvic tumor, and generate video generation parameters in combination with the user ID and the virtual physician ID; create the virtual physician's response video through the video generation platform, and then parse the video address and play it to the user.

[0083] The user can select a virtual doctor image, and each virtual doctor image will correspond to a virtual doctor ID; the virtual doctor video generation parameters are formed according to the reply content of the interactive conversation, the requesting user identifier and the virtual doctor ID, and the virtual doctor video generation parameters and the virtual doctor video generation request are sent to the virtual human video generation platform; the virtual human video generation platform generates relevant reply videos according to the virtual doctor video generation parameters, and returns the reply video address by calling the video generation interface; the reply video address is parsed according to the native video tag to obtain the reply video, and the virtual doctor reply video is played to the user.

[0084] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A female pelvic tumor knowledge dialogue question-answering method based on a large model, characterized in that: The method comprises: Step S1, after preprocessing the question text input by the user and extracting text features, the identified named entities are mapped, and the mapped named entities are used to perform female pelvic tumor knowledge retrieval results using the knowledge graph; Step S2: Use OCR to perform structured processing on the existing case data uploaded by the user, and use a large model to extract prescription text features from the key information identified by the processing; Step S3, constructing a large model of female pelvic tumors, and inputting the female pelvic tumor knowledge retrieval results and the prescription text feature extraction into the large model of female pelvic tumors for training, to obtain a trained large model of female pelvic tumors; Step S4: input the real-time multimodal data into the trained large model of female pelvic tumors to obtain the results of the female pelvic tumor knowledge dialogue question and answer.

2. The female pelvic tumor knowledge dialogue question-answering method based on a large model according to claim 1 is characterized in that: In step S1, after preprocessing the question text input by the user and extracting text features, the identified named entities are mapped, and the mapped named entities are used to perform female pelvic tumor knowledge retrieval results using the knowledge graph, specifically including: Use the bert-base-chinese model to extract features from the preprocessed question text to obtain question text features; Obtaining named entities based on the question text features, mapping the named entities, and converting the question text input by the user into machine language; The machine language is input into the knowledge graph for retrieval to obtain female pelvic tumor knowledge retrieval results.

3. The female pelvic tumor knowledge dialogue question-answering method based on a large model according to claim 2 is characterized in that: The construction process of the knowledge graph specifically includes: Preprocess the existing medical knowledge on female pelvic tumors to obtain preprocessed data; Using natural language processing to perform polarity analysis on the preprocessed data and extract entity, relationship and attribute information; The entities, relationships and attribute information are organized into a graph structure, and a knowledge graph is constructed based on the graph structure.

4. The female pelvic tumor knowledge dialogue question-answering method based on a large model according to claim 1 is characterized in that: In step S2, the process of using OCR to perform structured processing on the existing case data uploaded by the user and using a large model to extract prescription text features from the key information identified by the processing specifically includes: Use OCR to structure the existing case data uploaded by users to obtain structured data; Segmenting the structured data to obtain a number of word texts, and performing spelling check and correction on the word texts; Use a binary decision tree to clean and remove duplicates from the checked and corrected word text to obtain key information; The key information is used to extract prescription text features using a large model.

5. A female pelvic tumor knowledge dialogue question-answering system based on a large model, the knowledge dialogue question-answering system is used to implement the knowledge dialogue question-answering method described in any one of claims 1-4, characterized in that: The system comprises: The knowledge retrieval module is used to pre-process the question text input by the user and extract text features, then map the identified named entities, and use the knowledge graph to retrieve the knowledge results of female pelvic tumors based on the mapped named entities; The prescription text extraction module is used to use OCR to perform structured processing on the existing case data uploaded by the user, and use the large model to extract prescription text features from the key information identified; A large model question-answering module is used to construct a large model of female pelvic tumors, based on the female pelvic tumor knowledge retrieval results and the prescription text feature extraction input into the female pelvic tumor large model for training, to obtain a trained large model of female pelvic tumors; The virtual physician dialogue module is used to input real-time multimodal data into the trained large model of female pelvic tumors to obtain female pelvic tumor knowledge dialogue question and answer results.

6. The female pelvic tumor knowledge dialogue question-answering system based on a large model according to claim 5 is characterized in that: The workflow of the knowledge retrieval module specifically includes: Use the bert-base-chinese model to extract features from the preprocessed question text to obtain question text features; Obtaining named entities based on the question text features, mapping the named entities, and converting the question text input by the user into machine language; The machine language is input into the knowledge graph for retrieval to obtain female pelvic tumor knowledge retrieval results.

7. The female pelvic tumor knowledge dialogue question-answering system based on a large model according to claim 6 is characterized in that: The construction process of the knowledge graph specifically includes: Preprocess the existing medical knowledge on female pelvic tumors to obtain preprocessed data; Using natural language processing to perform polarity analysis on the preprocessed data and extract entity, relationship and attribute information; The entities, relationships and attribute information are organized into a graph structure, and a knowledge graph is constructed based on the graph structure.

8. The female pelvic tumor knowledge dialogue question-answering system based on a large model according to claim 5 is characterized in that: The workflow of the prescription text extraction module specifically includes: Use OCR to structure the existing case data uploaded by users to obtain structured data; Segmenting the structured data to obtain a number of word texts, and performing spelling check and correction on the word texts; Use a binary decision tree to clean and remove duplicates from the checked and corrected word text to obtain key information; The key information is used to extract prescription text features using a large model.

Images