An intelligent search system for the medical field

Abstract

The application discloses a medical field-oriented intelligent search system which is composed of a data management module, a data processing module, a natural language processing service module, a natural language understanding module, an information retrieval module and a result display module. The application realizes a search function based on natural language understanding by applying and researching Solr and NLU models and processing medical data by using a natural language processing technology. The search function has distinct intention characteristics of a patient's question sentence, which is the reason for selecting medical data. The NLU model data is trained by a large amount of medical data marking, so that the NLU model data has excellent intention classification characteristics in different departments and different types of diseases. To some extent, the application enables a search engine to understand a user's search intention more and makes the search engine more intelligent.

Description

Technical Field

[0001] This invention relates to an intelligent search system for medical data based on natural language understanding, belonging to the field of computer technology. Background Technology

[0002] In daily life, abundant information resources have impacted people's lives on various levels, achieving symbiosis and integration at the social level, subverting traditional thinking patterns, and laying the foundation for harmonious coexistence between humanity and nature. Digitalization has become a crucial force in building modern society. At the individual level, changes in our lifestyles, work methods, and learning methods have brought us convenience and speed. However, the sheer volume of information resources online makes retrieving usable information from this rich and diverse data a challenge. Traditional search engines can directly help us find the information and resources we need, but the accuracy, speed, and relevance of information retrieval cannot adequately meet user needs. According to Smartinsight, there are currently 5 billion searches globally every day, with 3.5 billion of them coming from Google, accounting for 70% of global searches, equivalent to processing more than 40,000 searches per second.

[0003] The working principle of traditional search engines mainly includes web crawling, database building, web page processing, information retrieval, and result ranking and display. Google, which became one of the best search engines on the World Wide Web, was initially a prototype system implemented by Stanford University doctoral students Sergey Brin and Lawrence Page. Google's architecture is similar to traditional search engines, but its biggest difference lies in its ranking of web pages based on authority values, ensuring that the most important pages appear at the top of the results.

[0004] Google calculates a webpage's PageRank value using a meta-algorithm, which determines the page's position in the results set; a higher PageRank value places the page higher in the results. While this workflow can largely solve users' search needs, it suffers from problems such as large amounts of information in search results and the inability of the returned content to accurately understand user intent. These issues pose significant challenges to traditional search engines. However, the current Google search engine utilizes artificial intelligence technologies such as knowledge graphs and natural language understanding to optimize search results, making the search engine more "human-like."

[0005] Users desire more convenient and faster access to search services, and more accurate results. Intelligent search has emerged to address these needs. Intelligent search includes knowledge graph-based intelligent search services and natural language processing (NLP)-based intelligent search services. These services incorporate artificial intelligence (AI) applications, making traditional search engines more intelligent and "human-like." As a result, intelligent search with AI-powered NLP has become a popular and important research direction. The application of NLP and natural language understanding technologies to search engines allows for a more accurate understanding of user search intent. Deep learning and machine learning techniques can convert natural language into a vector-like representation that machines can understand, which is then used for model training. Intent keywords and entity keywords are extracted from the natural language query and added to the NLP module, enabling interaction between natural language and the search engine. Users can then experience a more accurate, timely, and efficient search experience. Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent search solution based on natural language understanding, and to design and develop an intelligent perception search system based on natural language understanding for question and answer data related to medical consultations.

[0007] The technical solution adopted in this invention is an intelligent search system for the medical field, which consists of the following modules: a data management module, a data processing module, a natural language processing service module, a natural language understanding module, an information retrieval module, and a results display module. The data management module and the data processing module input the processed data into the information retrieval module. The information retrieval module interacts with the natural language processing service module, interacts with the natural language understanding module, and is connected to the results display module.

[0008] This invention describes the entire system architecture through the following steps, which facilitates understanding of the entire system's process and the relationships between modules.

[0009] Step 1: Data Management;

[0010] First, a Solr collection is created using the Solr data management module to store medical data. The local medical data is in CSV format. The data is read from the CSV file and converted to the correct format for subsequent indexing.

[0011] After creating the collection, configure the content fields within the collection. This is because when indexing data later, the fields existing in the medical data need to be predefined. Only after all existing fields are predefined can the medical data with the required response fields be successfully imported into the Solr collection.

[0012] Step 2: Data processing;

[0013] Local medical data is stored in CSV format. Before indexing this data into a Solr collection, the data needs to be cleaned and filtered accordingly, including short text filtering, duplicate text filtering, and invalid text filtering.

[0014] Because this data is crawled from the web, it contains duplicate text and redundant invalid text. If this text is not processed before indexing, it will affect search results and data evaluation. For example, after crawling the data, it is necessary to classify and categorize it into the corresponding attribute fields when saving the data. Different processing techniques should be used for patient questions, doctor answers, and the department to which the data belongs to achieve the corresponding processing requirements.

[0015] Step 3: Information Retrieval;

[0016] The interaction of information retrieval lies in the establishment of a searchable interface between the user and Solr data, and it supports two search modes, including general search based on keywords and search based on natural language understanding.

[0017] This module provides users with an input search interface. After entering a query in natural language, users can choose whether to enable the intent recognition module stage through the Pipeline configuration. If this module is not enabled, it is processed through natural language processing, which segments the query into words using Chinese word segmentation, obtains keywords, and then directly performs a related search on the entire database to obtain search results. The effect is poor; it can basically meet the user's search needs, but it cannot well understand the user's true search intent.

[0018] If the intent recognition module is enabled, the corresponding search algorithm template needs to be configured in the intent recognition stage. After the user inputs a natural language query, the intent recognition module converts the unstructured natural language search statement into structured JSON format data, including medical entities in the statement, model-predicted intent keywords, and medical keywords. This data is used to populate the subsequent search algorithm template. Compared to ordinary keyword-based searches, natural language understanding-based search better understands the user's true search intent and returns more accurate search results. Instead of simply using keywords to retrieve all data, it improves both search efficiency and accuracy.

[0019] Step 4: Results Display;

[0020] The two types of search results are displayed differently. Keyword-based search results only include the search results, search time, and ranking score. Natural Language Understanding-based search results, in addition to the results from the regular search, also include results generated by the intent recognition stage. These results can be modified and saved as training data for the NLU model, used to optimize the model and improve its intent recognition capabilities.

[0021] In summary, this invention, through the application and research of Solr and NLU models, and the use of natural language processing (NLP) technology to process medical data, achieves a search function based on natural language understanding. This is achieved by utilizing the clear intent characteristics of patients' interrogative statements, which is also the reason for selecting medical data. By training the NLU model with a large amount of labeled medical data, it exhibits excellent intent classification characteristics across different departments and types of diseases. This invention, to a certain extent, enables search engines to better understand users' search intent, making search engines more intelligent. Attached Figure Description

[0022] Figure 1 This is a diagram of an intelligent sensing search system and its sub-modules.

[0023] Figure 2 A diagram illustrating the configuration file for model training.

[0024] Figure 3 This is a flowchart of the Natural Language Understanding module.

[0025] Figure 4 A template diagram for a custom search algorithm.

[0026] Figure 5 This is a flowchart of the natural language search process.

[0027] Figure 6 This is a structural diagram of an intelligent sensing and search system. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] This invention utilizes natural language query processing techniques, including word segmentation, named entity recognition, and part-of-speech tagging, combined with the intent recognition technology of the underlying search engine Solr and Rasa_NLU, as well as the Spring Boot framework and Python resource packages, to develop an intelligent perception search system that can "humanize" the user's search intent.

[0030] From a functional perspective, this system can be divided into the following six sub-modules: data management module, data processing module, natural language processing module, natural language understanding module, information retrieval module, and results display module. Its organizational structure is as follows: Figure 1 As shown.

[0031] The data management module includes the integration of medical data, encompassing six major medical specialties: andrology, internal medicine, obstetrics and gynecology, oncology, pediatrics, and surgery. This data is integrated and stored in Solr collections through Solr schema configuration files. Solr's data management primarily handles the creation, configuration, and deletion of dataset collections. The configuration, addition, and deletion of collection fields form the raw dataset supporting subsequent user queries. Collection fields include question fields, answer fields, intent fields, ranking scores, and department fields.

[0032] The data processing module cleans all data, defines relevant intent keywords for different departments, and categorizes real patient consultation questions within each department. Patient consultation questions can be labeled with different medical domain entities, such as medical location, time of onset, and disease type. Before being stored in Solr's collection, the raw data undergoes processing, including short text filtering, duplicate text filtering, and invalid text filtering. This data serves as the original dataset for queries and the training dataset for model training, providing data support for the natural language processing and natural language understanding modules, ensuring that all data to be indexed is scalable and possesses a certain level of high value.

[0033] Another important function of the data processing module is to configure the search pipeline. By creating a pipeline name and configuring the corresponding stage in the pipeline, this system can realize the search function based on natural language understanding by configuring the custom search algorithm template required by the intent recognition algorithm.

[0034] Natural Language Processing Module: The Natural Language Processing module primarily processes user queries. It processes users' natural language search statements by configuring technologies such as disabling dictionaries and word segmentation methods. It uses Solr to configure Ansj word segmentation for basic segmentation of query statements, and then further improves the accuracy of search results by using the Natural Language Understanding module.

[0035] Natural Language Understanding (NLP) Module: While regular expressions or template matching can be used to understand user search behavior, issues such as decreased search result accuracy and inability to truly understand user intent persist. The main function of the NLP module is to understand the true intent of the user's search and extract relevant medical entities from the user's search query for further searching. Using intent keywords and medical entity keywords can improve search accuracy. Specific configuration is as follows: The NLP module can be configured with different pipelines, and its backend implementation supports spaCy, MITIE, sklearn, tensorflow, etc.

[0036] The pipeline used in this experiment is MITIE+Jieba+sklearn, and the configuration file is config_jieba_mitie_sklearn.yml, as follows: Figure 2 As shown. The sklearn+MITIE configuration is used because sklearn has fast and good intent classification capabilities, and MITIE can generate good feature vectors and entity recognition. Jieba segmentation is used because it has good segmentation performance among all Chinese word segmenters. In summary, the MITIE+Jieba+sklearn configuration pipeline is used.

[0037] The MITIE model used in the Pipeline was obtained through unsupervised model training. Since training the model is time-consuming and memory-intensive, the MITIE model used in this experiment was generated from open-source Chinese Wikipedia and Baidu Encyclopedia.

[0038] The medical data stored in the Solr collection and the obtained MITIE word vector model are used to train the NLU model. However, the medical data needs to be labeled. Labeling can be done manually or by using a data labeling platform.

[0039] Next, an NLU model can be trained using labeled medical data and word vector models. This model will be used for subsequent intent recognition. Model training can be performed offline or online in real-time. The trained model file will then be used to launch the intent recognition service. The service will be provided to the system via a web service interface using the Python-based Flask framework, and the return results will be encapsulated.

[0040] In summary, the entire process of the natural language understanding module is as follows: Figure 3 As shown, the results generated by calling the intent recognition interface include intent keywords and medical entities, and the configuration work for the user's subsequent search algorithm.

[0041] Information retrieval module: The information retrieval module is divided into general search and natural language understanding-based search, mainly based on medical data from Solr collections.

[0042] Regular search is based on keyword matching (using the Chinese word segmenter configured in the Solr search engine) and the query parser encapsulated in Solr to perform the search function.

[0043] Natural language understanding-based search involves combining user queries with intent keywords and medical entity keywords generated by an intent recognition service interface, along with a custom search algorithm, to form a query. Figure 4 As shown, the query statement is used in Solr's query parser to implement a search function based on natural language understanding.

[0044] Natural language search includes both of the above methods for fulfilling data search needs, such as Figure 5 As shown. The specific process is as follows:

[0045] (1) Search Process Based on Natural Language Understanding: First, a data management pipeline is created. The data management pipeline allows for the configuration of different stages to manipulate data. Here, a natural language understanding stage is configured, and a search algorithm template is written in the stage. Next, it is determined whether to add or enable this stage. If the intent recognition stage is enabled, three parts of content are generated through the natural language understanding module, including intent keywords, medical entities, and medical keywords. Finally, the keywords in the search template are replaced with the above content and recombined for use in the search engine to generate the final search results, thus completing the search function based on natural language understanding.

[0046] (2) Keyword-based normal search process: If the intent recognition stage is turned off, or if it is not used or not created, the default keyword-based search is used. The keywords of the user's search statement generated by the natural language processing module are used in the Solr search engine to generate the final search results and complete the keyword-based normal search function.

[0047] The results display module shows the total number of search results, search time, and ranking score of each search result. The highest-ranked result is displayed first, and all search results are displayed in descending order. Natural language understanding-based search, in addition to the search results from the regular search, also includes results from the natural language understanding service interface, and allows for modification. Intent recognition results include intent keywords, medical entity keywords, and medical entities (values). If the recognition results are incorrect, these results can be modified and saved as training data for the intent recognition model, further optimizing the model's accuracy and improving its intent recognition capabilities.

Claims

1. A medical field-oriented intelligent search system, characterized by, Comprise: data management module, data processing module, natural language processing service module, natural language understanding module, information retrieval module, result display module; the data management module and data processing module will process the input into the information retrieval module, information retrieval module and natural language processing service module interact with each other, information retrieval module and natural language understanding module interact with each other, information retrieval module and result display module are connected; The whole system architecture is described by the following steps, which is convenient for understanding the flow of the whole system and the connection between modules; Step 1: data management of data management module; Firstly, through the Solr data management module, the collection of Solr is created to store medical data. The local medical data is a csv format file. The data format conversion is carried out by reading the csv file data for subsequent index data work; After creating the collection, the content field of the collection is configured, and the required response field medical data is imported into the collection of Solr; Step 2: data processing of data processing module; The local medical data is stored in the form of csv. Before indexing the local medical data into the collection of Solr, the data needs to be cleaned and the response filtering processing, including short text filtering, repeated text filtering and invalid text filtering; Step 3: information retrieval of information retrieval module; The interaction of information retrieval is that the user and Solr data build a searchable interface, and support two search modes, including ordinary search based on keywords and search based on natural language understanding; The information retrieval module provides an inputtable search interface for users. After the user inputs the natural language query statement, the configuration Pipeline is used to select whether to enable the intent recognition module stage. If this module is not enabled, the natural language processing is carried out, the query statement is segmented by Chinese word segmentation, the keywords are obtained, and the related search of the whole database data is carried out directly, and the search result is obtained; If the intent recognition module is started, the response search algorithm template needs to be configured in the intent recognition stage. After the user inputs the natural language query statement, the unstructured natural language search statement is converted into structured json format data by the intent recognition module, including medical entities in the statement, model predicted intent keywords and medical keywords; These intent keywords and medical keywords are filling data in the subsequent search algorithm template; The search based on natural language understanding is more accurate in returning search results than the ordinary search based on keywords, which understands the real search intent of the user and is more accurate in returning search results; Step 4: result display of result display module; The two search result displays are different. The result display of the general search based on keywords only includes search results, search time consumption, and result ranking; the search result display based on natural language understanding includes the results generated by the intent recognition stage in addition to the results of the general search, and can be modified and saved as data for NLU model training to optimize the model and improve the intent recognition ability of the model; the data processing module also has an important function of configuring the Pipeline of the search, which realizes the search function based on natural language understanding by creating a Pipeline name, configuring the response stage in the Pipeline, and configuring the custom search algorithm template required by the intent recognition algorithm to search.

2. The intelligent search system for medical field according to claim 1, wherein, The data management module includes the integration of medical data, which contains six major categories: male, internal medicine, obstetrics and gynecology, oncology, pediatrics, and surgery. The data is integrated and stored by configuring the schema file of Solr so that all data is indexed into the collection of Solr; the data management of Solr is responsible for the creation, configuration, and deletion of the data set collection, and the configuration, addition, and deletion of the collection field, which is the original data set supporting subsequent user queries; The collection field includes the question field, the answer field, the intent field, the ranking score, and the department field. 3.The medical field-oriented intelligent search system according to claim 1, wherein, The data processing module cleans all data, defines related intent keywords in different departments, and classifies patient real consultation questions in each department. The patient's consultation questions can be labeled with different medical field entities. The original data needs to be processed before being stored in the collection of Solr, so that all indexed data has scalability and certain high value.

4. The intelligent search system for the medical field according to claim 1, characterized in that, The natural language processing service module processes the user's query statement, processes the user's natural language search statement by configuring a stop word dictionary and a word segmentation method, uses Solr to configure Ansj word segmentation for basic segmentation of the query statement, and further improves the accuracy of the search result by using the natural language understanding module.

5. The intelligent search system for medical field according to claim 1, wherein, The natural language understanding module understands the true intent of the user's search and extracts corresponding medical entities for searching from the user's search question, improves the accuracy of the search by using intent keywords and medical entity keywords, and is specifically configured as follows: the natural language understanding module supports spaCy, MITIE, sklearn, and tensorflow through different Pipeline configurations.

6. The intelligent search system for medical field according to claim 1, wherein, Information retrieval module: The information retrieval module is divided into general search and search based on natural language understanding, which is based on the medical data of the Solr collection; General search refers to keyword matching and search function realized by the query parser encapsulated by Solr. The natural language understanding-based search is to generate the intent keyword and the medical entity keyword by calling the intent recognition service interface with the user query sentence, to form a query sentence with the self-defined search algorithm, to use the query sentence in the query parser of Solr, and to realize the natural language understanding-based search function.

7. The intelligent search system for medical field according to claim 1, wherein, The natural language search includes the above two ways of searching data, and the specific process is as follows: (1) The natural language understanding-based search process: first, create a data management Pipeline, the function of the data management Pipeline is to configure different stages to operate the data, here configure the natural language understanding stage, write the search algorithm template in the stage, then judge whether to add to start the stage, if the intent recognition stage is started, generate three parts of content through the natural language understanding module, including the intent keyword, the medical entity, and the medical keyword; finally, replace the keywords in the search template to reorganize, used in the search engine, to produce the final search result, to complete the natural language understanding-based search function; (2) The keyword-based ordinary search process: if the intent recognition stage is closed, create a situation that is not used or not created, then use the default search based on the keyword, use the keyword of the user search sentence generated by the natural language processing module in the Solr search engine, to produce the final search result, to complete the keyword-based ordinary search function. 8.The medical field-oriented intelligent search system according to claim 1, wherein, The result display module: the user's ordinary search result display includes the total number of search results, search time consumption, and search result ranking score; The one with the highest ranking score is displayed first, and all search results are displayed in descending order; the natural language understanding-based search includes the search results of the ordinary search and the result display and modification function of the natural language understanding service interface; the results of the intent recognition include the intent keyword intent, the medical entity keyword entity, and the medical entity value, if the recognition result is wrong, these results can be modified and saved as the intent recognition model training data, to optimize the accuracy of the model and improve the intent recognition ability of the model.

Images