Domain knowledge extraction method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides a domain knowledge extraction method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining a domain entity and an entity relationship of a target domain extracted by a large model from dialogue data questioned by a user based on a cue word; the target domain is determined based on dialogue data questioned by the user; calculating the confidence of each domain entity based on the domain dictionary of the target domain; the domain dictionary is acquired according to accumulation of historical dialogue data of the large model in a target domain; and screening the domain entities according to the confidence, and updating the knowledge graph of the target domain according to the screened high-confidence target entities and the corresponding entity relationships thereof. By guiding the large model to extract the entities and the entity relationship and screening the extracted entities based on the confidence, dynamic synchronization of domain knowledge and business requirements is ensured, the knowledge extraction efficiency is improved, the knowledge updating period is shortened, and the business requirements of dynamic changes are met.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and program product for extracting domain knowledge. Background Art

[0002] With the widespread application of big models in various fields, the importance of extracting domain knowledge has become increasingly prominent. As generative dialogue models, big models rely on the construction of knowledge bases. They provide users with the information they need by matching keywords against the knowledge base or implementing question-and-answer services through rule engines. Currently, the knowledge bases of big models mainly use manual annotation to construct knowledge graphs. This reliance on manual annotation for extracting domain knowledge is inefficient, has limited coverage, and has a long knowledge update cycle, making it impossible to synchronize with business changes or respond to business needs in real time. Summary of the Invention

[0003] The present application provides a domain knowledge extraction method, apparatus, equipment, storage medium and program product to solve the defects of the prior art in constructing large-model knowledge graphs through manual annotation, which are inefficient, have limited coverage, have a long knowledge update cycle, and cannot synchronize business changes or respond to business needs in real time.

[0004] This application provides a domain knowledge extraction method, including: Obtaining domain entities and entity relationships of a target domain; the domain entities and entity relationships are extracted by a large model based on prompt words from conversation data of user questions, and the target domain is determined based on the conversation data; Calculating the confidence of each entity in the domain based on a domain dictionary of the target domain; the domain dictionary is accumulated based on historical conversation data of the large model in the target domain; Screening the domain entities according to the confidence levels to obtain target entities with high confidence levels; Based on the target entity and the entity relationship corresponding to the target entity, the knowledge graph of the target domain is updated.

[0005] In one embodiment, the calculating the confidence of each domain entity based on the domain dictionary of the target domain includes: Based on the domain dictionary of the target domain, counting the word frequency and inverse document rate of the domain entity in the domain dictionary; Determining a domain weight of the domain entity in the target domain according to the word frequency and the inverse document rate; Obtaining entity extraction accuracy of the target domain; the entity extraction accuracy is determined based on a ratio of the number of entities in the target domain that are verified to be correctly extracted to the total number of entities extracted; The confidence of the domain entity is determined according to the entity extraction accuracy and the domain weight.

[0006] In one embodiment, the counting of the word frequency and inverse document rate of the domain entity in the domain dictionary based on the target domain includes: Based on the domain dictionary of the target domain, counting the frequency of occurrence of the domain entity in the domain dictionary; Determining the word frequency of the domain entity in the domain dictionary according to the occurrence frequency; Acquire a first document quantity of domain documents included in the domain dictionary, and a second document quantity of target documents including the domain entity in the domain documents; An inverse document rate of the domain entity in the domain dictionary is determined according to a ratio of the second document quantity to the first document quantity.

[0007] In one embodiment, updating the knowledge graph of the target domain based on the target entity and the entity relationship corresponding to the target entity includes: Calculating the association strength between a first target entity and each second target entity; the first target entity is any one of the target entities, and the second target entity is any one of the domain entities of the target entity and the knowledge graph of the target domain, except the first target entity; The target entity is used as a node, the entity relationship corresponding to the target entity is used as an edge, the association strength is used as the edge weight of the edge, and the knowledge graph of the target domain is updated.

[0008] In one embodiment, calculating the association strength between the first target entity and the second target entity includes: Counting the number of co-occurrences of the first target entity and each second target entity in the target domain to obtain a co-occurrence frequency of the first target entity and the second target entity; Normalizing the co-occurrence frequency based on the maximum co-occurrence frequency corresponding to the target domain to obtain a normalized co-occurrence frequency; Vectorizing the first target entity and the second target entity to obtain a first entity vector of the first target entity and a second entity vector of the second target entity; Calculating the semantic similarity between the first target entity and the second target entity based on the first entity vector and the second entity vector; A weighted sum is performed on the normalized co-occurrence frequency and the semantic similarity to obtain the association strength between the first target entity and the second target entity.

[0009] In one embodiment, after updating the knowledge graph of the target domain based on the target entity and the entity relationship corresponding to the target entity, the method further includes: Obtaining the indicator value of the reward indicator corresponding to the response data of the large model; the reward indicator includes response time, answer accuracy and user score; the response data is generated when the large model responds to the dialogue data; Based on the indicator value and the reward function, a reward value of the reasoning path corresponding to the response data is calculated; the reasoning path is a path for performing knowledge retrieval on the domain knowledge graph according to the association strength between each domain entity in the domain knowledge graph when the response data is generated by the large model; the reward function is constructed based on the reward indicator; The reasoning path is optimized according to the reward value.

[0010] This application also provides a domain knowledge extraction device, including the following modules: An entity extraction module is used to obtain domain entities and entity relationships of a target domain; the domain entities and entity relationships are extracted from user question conversation data by a large model based on prompt words, and the target domain is determined based on the conversation data; A confidence calculation module, configured to calculate the confidence of each entity in the domain based on a domain dictionary of the target domain; the domain dictionary is accumulated based on historical conversation data of the large model in the target domain; An entity screening module, configured to screen the domain entities according to the confidence level to obtain target entities with high confidence levels; A domain knowledge updating module is used to update the knowledge graph of the target domain based on the target entity and the entity relationship corresponding to the target entity.

[0011] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the domain knowledge extraction method as described above is implemented.

[0012] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the domain knowledge extraction method as described above is implemented.

[0013] The present application also provides a computer program product, comprising a computer program, which implements any of the above-described domain knowledge extraction methods when executed by a processor.

[0014] The domain knowledge extraction method, device, equipment, storage medium and program product provided in this application, by guiding the large model to extract entities and relationships from the unstructured conversation data of user questions, provides a structured data foundation for the construction of the knowledge graph, replaces the manual annotation method to extract domain knowledge, ensures the dynamic and coverage of domain knowledge extraction, and improves the efficiency of domain knowledge extraction and update. In addition, the confidence of the extracted entities is calculated through the domain dictionary, and high-confidence entities and their corresponding entity relationships are screened out to update the domain knowledge graph. While ensuring the accuracy of the domain knowledge, it realizes the dynamic update of the domain knowledge, shortens the knowledge update cycle, and is conducive to the dynamic integration of domain knowledge and business needs, thereby being able to meet dynamically changing business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 It is a flowchart of the domain knowledge extraction method provided in the embodiment of the present application.

[0017] Figure 2 It is a structural diagram of the domain knowledge extraction device provided by the present invention.

[0018] Figure 3 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0020] An embodiment of the present application provides a method for extracting domain knowledge, which uses prompt words to guide a large model to extract domain entities from conversation data with users, dynamically updates the knowledge graph in real time, and realizes automatic extraction and updating of domain knowledge, thereby improving the efficiency of domain knowledge extraction. It can also update domain knowledge in real time and synchronize business changes to respond to business needs in a timely manner.

[0021] Specifically, Figure 1This is a flow chart of the domain knowledge extraction method provided in the embodiment of the present application. Figure 1 As shown, the method includes the following steps: Step 100: Acquire domain entities and entity relationships of a target domain; the domain entities and entity relationships are extracted from user question conversation data by a large model based on prompt words, and the target domain is determined based on the conversation data; Step 200: Calculate the confidence of each entity in the domain based on the domain dictionary of the target domain; the domain dictionary is accumulated based on the historical conversation data of the target domain by the large model; Step 300, screening the domain entities according to the confidence level to obtain target entities with high confidence level; Step 400: Generate or update the knowledge graph of the target domain based on the target entity and the entity relationship corresponding to the target entity.

[0022] First, the domain entities and entity relationships of the target domain are obtained. These domain entities and entity relationships are extracted by the large model from user conversation data based on prompt words. The target domain is determined based on the large model's conversation data. Prompt words instruct the large model to extract entities and entity relationships from the unstructured conversation data of user questions, providing a structured data foundation for building the knowledge graph.

[0023] For example, the extracted domain entities include “product model” and “fault type”, and the extracted entity relationships include “cause” and “solution”.

[0024] Optionally, during a conversation with a user, the large model performs intent recognition on the conversation data from the user's questions to determine the user's intent. Based on the user's intent, the large model can determine the domain corresponding to the conversation data. Guided by prompt words, the large model extracts entities and relationships between entities in the corresponding domain from the conversation data from the user's questions, thereby obtaining the domain entities and entity relationships extracted by the large model.

[0025] The confidence level of each domain entity is calculated based on the domain dictionary of the target domain. The domain dictionary of the target domain is accumulated from the historical conversation data of the large model in the target domain. The domain dictionary of the target domain is constructed by accumulating historical conversation data of users asking questions in the target domain and combining it with existing domain knowledge in the target domain.

[0026] The confidence of a domain entity represents the probability that the domain entity belongs to the target domain, that is, the credibility of the relationship between the domain entity and the target domain. The same entity can belong to multiple different domains, and the confidence of the same entity in different domains can be the same or different. The same domain can contain multiple domain entities, and the confidence of each domain entity can be the same or different.

[0027] Based on the confidence level of each extracted domain entity, the extracted domain entities are screened to obtain high-confidence target entities. By calculating the confidence level of each domain entity, high-confidence entities and low-confidence entities are screened. Low-confidence entities may be noise or incorrectly extracted entities. For example, if the confidence level of an entity (such as "fault code") is 0.9, it indicates that the entity is highly likely correct; if the confidence level is 0.3, it is likely noise or incorrectly extracted. Therefore, by screening high-confidence entities based on confidence level, we can prevent low-quality data from contaminating the knowledge base of the large model and ensure the credibility and accuracy of domain knowledge.

[0028] Furthermore, based on the selected high-confidence target entities and the entity relationships corresponding to the target entities, the knowledge graph of the target domain is updated, thereby updating the extracted domain knowledge to the knowledge graph for storage, completing the extraction of domain knowledge. Optionally, the target domain can be a new domain or an existing domain. If the target domain is a new domain, the knowledge graph of the target domain is constructed based on the existing initial domain knowledge of the target domain. If the target domain is an existing domain, the knowledge graph of the target domain is constructed based on the existing initial domain knowledge of the target domain and updated based on the domain entities and entity relationships extracted from the large model.

[0029] Updating the knowledge graph includes adding the target entity and its corresponding entity relationship to the domain knowledge graph, or replacing the same entity in the knowledge graph. Specifically, first obtain the knowledge graph of the target domain, then perform entity retrieval on the knowledge graph based on the target entity to determine whether the target entity already exists in the knowledge graph of the target domain; if so, determine the entity relationship between the target entity and other entities, and update the existing entity relationship in the domain knowledge graph based on the entity relationship corresponding to the target entity; if not, add the target entity and its corresponding entity relationship to the existing domain knowledge graph.

[0030] When updating the knowledge graph of the target domain, it is also necessary to determine the entity relationship between the target entity and entities in other domains in the knowledge graph, establish a connection between the target entity and other entities in the knowledge graph based on the entity relationship, add the target entity, and complete the update of the knowledge graph.

[0031] In this embodiment, by guiding the large model to extract entities and relationships from the unstructured conversational data of user questions, a structured data foundation is provided for the construction of the knowledge graph, replacing manual annotation to extract domain knowledge. This ensures the dynamic nature and coverage of domain knowledge extraction, and improves the efficiency of domain knowledge extraction and updating. Furthermore, by calculating the confidence of the extracted entities using a domain dictionary, high-confidence entities and their corresponding entity relationships are screened, and the domain knowledge graph is updated. This ensures the accuracy of domain knowledge while enabling dynamic updating of domain knowledge, shortening the knowledge update cycle, and facilitating the dynamic integration of domain knowledge and business needs, thereby meeting dynamically changing business needs.

[0032] In one embodiment, the confidence of a domain entity is calculated based on the weight of the target entity in the domain by using the domain dictionary and combined with the extraction accuracy of the entity in the domain. Specifically, step 200 may include: Step 210: Based on the domain dictionary of the target domain, count the word frequency and inverse document rate of the domain entity in the domain dictionary; Step 220: determining the domain weight of the domain entity in the target domain based on the word frequency and the inverse document rate; Step 230: Obtaining the entity extraction accuracy of the target domain; the entity extraction accuracy is determined based on the ratio of the number of entities verified to be correctly extracted in the target domain to the total number of entities extracted; Step 240: Determine the confidence of the domain entity based on the entity extraction accuracy and the domain weight.

[0033] When calculating the confidence of a domain entity, we first count the word frequency and inverse document ratio of the domain entity in the domain dictionary of the target domain. Based on these counts, we determine the domain weight of the domain entity in the target domain. Next, we obtain the entity extraction accuracy of the target domain. This accuracy is determined by the ratio of the number of verified correctly extracted entities to the total number of extracted entities in the target domain. Finally, we determine the confidence of the domain entity based on the entity extraction accuracy of the target domain and the calculated domain weight.

[0034] Optionally, the domain weight of the domain entity in the target domain represents its importance in the target domain. This domain weight is calculated based on the domain entity's term frequency and inverse document rate in the domain dictionary. The term frequency and inverse document rate are calculated using the TF-IDF (Term Frequency–Inverse Document Frequency) algorithm. Based on this, step 210 may further include: Step 211: Based on the domain dictionary of the target domain, count the occurrence frequencies of the domain entities in the domain dictionary; Step 212, determining the word frequency of the domain entity in the domain dictionary according to the occurrence frequency; Step 213: obtaining a first number of domain documents included in the domain dictionary, and a second number of target documents including the domain entity in the domain documents; Step 214 : Determine the inverse document rate of the domain entity in the domain dictionary according to the ratio of the second document quantity to the first document quantity.

[0035] For word frequency, based on a domain dictionary of the target domain, the frequency of occurrence of domain entities in the domain dictionary is counted. Based on the frequency of occurrence, the word frequency of the domain entity in the domain dictionary is determined. Furthermore, the domain dictionary includes multiple domain documents. For inverse document frequency, a first number of domain documents included in the domain dictionary and a second number of target documents containing the domain entity in the domain documents are obtained. Based on the ratio of the second number of documents to the first number of documents, the inverse document rate of the domain entity in the domain dictionary is determined.

[0036] In one embodiment, for any domain entity , its confidence It can be calculated as shown in the following formula 1: ; (1) in, is the number of entities that are verified to be correctly extracted in the target domain, is the total number of entities extracted in the target domain, It is a time decay factor used to reduce the weight of historical data and ensure that the large model pays more attention to the latest business data. and The ratio of the extracted entities to the extracted entities represents the entity extraction accuracy for the target domain. For example, if the large model extracts 100 entities and 80 are verified to be correct, the entity extraction accuracy is 0.8. This ratio directly reflects the large model's current accuracy in extracting domain entities and can be updated regularly to adapt to large model iterations. Verification of the correctness of extracted entities can be manual or through the large model, with no specific restrictions.

[0037] By combining domain relevance and time decay, the entity confidence can be dynamically adjusted. Based on this, the confidence of entities in the same domain extracted by the large model in different time periods can be the same or different in the same domain.

[0038] If the accuracy of entity extraction after the big model is updated is improved, the confidence of the subsequently extracted domain entities will automatically increase. From the perspective of business adaptation, when the business focus changes, for example, from "package consultation" to "fault handling", the confidence of the corresponding entity will change accordingly after the domain weight is adjusted, so that business changes can be synchronized in real time and dynamically to respond to business needs.

[0039] Is a domain entity The domain weight in the target domain can be obtained based on the TF-IDF algorithm. Specifically, the domain weight is calculated by counting the term frequency (TF) and inverse document frequency (IDF) of the domain entity in the domain dictionary based on the TF-IDF algorithm, and the following formula 2 is used to calculate the domain weight: ; (2) In formula 2, Representing domain entities The word frequency statistics in the domain dictionary of the target domain, is the total number of domain documents contained in the domain dictionary, Indicates that the domain dictionary contains domain entities The number of domain documents, and Taking the logarithm of the ratio of , we get the inverse document rate of the domain entity in the domain dictionary. In other words, the domain weight of the domain entity in the target domain is the product of the domain entity's word frequency in the domain dictionary of the target domain and the inverse document rate.

[0040] In practical applications, when updating entity extraction accuracy, for example, due to the complexity of text semantics and semantic scenarios, some high-confidence entities may be marked as erroneous entities after verification. For example, "apple" can refer to a mobile phone brand, not a fruit, in certain scenarios. Domain entities marked as erroneous after verification can be corrected manually or through other means to update the accuracy of entity extraction by the large model. To update business data, during the user's conversation with the large model, new domain documents are generated based on the conversation data, and the entity's domain weight is dynamically updated using the TF-IDF algorithm, thereby optimizing the confidence of subsequently extracted entities.

[0041] By dynamically adjusting entity confidence, domain knowledge extraction can be self-iterated, reducing manual intervention costs. Entity confidence, serving as a quality metric for large-scale model entity extraction results, quantifies entity extraction accuracy, domain relevance based on domain dictionaries, and timeliness based on time decay factors. This ensures that only high-value entities are extracted from large-scale models, providing a reliable data foundation for knowledge graph construction and the large-scale model's inference engine. The dynamic update feature also enables large-scale models to autonomously optimize and adapt to business changes.

[0042] Optionally, the domain knowledge graph is constructed with entities as nodes, entity relationships as edges between nodes, and the strength of association between entities as edge weights. The strength of association between entities is determined based on the co-occurrence of entities. Therefore, step 300 may also include: Step 310: Calculate the association strength between the first target entity and each second target entity; the first target entity is any one of the target entities, and the second target entity is any one of the domain entities in the target entity and the knowledge graph of the target domain, except the first target entity; Step 320 , updates the knowledge graph of the target domain with the target entity as a node, the entity relationship corresponding to the target entity as an edge, and the association strength as the edge weight of the edge.

[0043] Based on the filtered target entities, the association strength between the first target entity and each second target entity is calculated, and the knowledge graph of the target field is updated with the target entity as the node, the entity relationship corresponding to the target entity as the edge, and the calculated association strength as the edge weight.

[0044] Among them, the first target entity is any one of the screened target entities, and the second target entity is any one of the screened target entities and the domain entities in the knowledge graph of the target domain, except the first target entity.

[0045] For the first target entity, calculate its association strength with other target entities and each domain entity in the knowledge graph, using this as the weight of the entity relationship between the entities. Based on this, update the first target entity to the knowledge graph. In this way, each target entity is updated to the knowledge graph, completing the extraction of domain knowledge.

[0046] In one embodiment, the association strength between entities is determined based on the co-occurrence frequency between the entities and the semantic similarity between the entities. Therefore, step 310 may further include: Step 311: Count the number of co-occurrences of the first target entity and each second target entity in the target domain to obtain the co-occurrence frequency of the first target entity and the second target entity; Step 312: normalize the co-occurrence frequency based on the maximum co-occurrence frequency corresponding to the target domain to obtain a normalized co-occurrence frequency; Step 313: vectorize the first target entity and the second target entity to obtain a first entity vector of the first target entity and a second entity vector of the second target entity; Step 314: Calculate the semantic similarity between the first target entity and the second target entity based on the first entity vector and the second entity vector; Step 315 : Perform weighted summation on the normalized co-occurrence frequency and the semantic similarity to obtain the association strength between the first target entity and the second target entity.

[0047] When calculating the association strength between the first target entity and the second target entity, the number of co-occurrences of the first target entity and each second target entity in the target domain is first counted to obtain the co-occurrence frequency of the first target entity and the second target entity. Based on the maximum co-occurrence frequency corresponding to the target domain, the co-occurrence frequency of the first target entity and the second target entity is normalized to obtain the normalized co-occurrence frequency.

[0048] It should be noted that the number of co-occurrences of the first target entity and the second target entity in the target domain specifically refers to the number of times the first target entity and the second target entity co-appear in the historical conversations of the large model.

[0049] Then, the first target entity and the second target entity are vectorized to obtain a first entity vector of the first target entity and a second entity vector of the second target entity. The semantic similarity between the first target entity and the second target entity is calculated based on the first entity vector and the second entity vector.

[0050] Finally, the normalized co-occurrence frequency and semantic similarity between the first target entity and the second target entity are weighted and summed to obtain the association strength between the first target entity and the second target entity.

[0051] Optionally, the first and second target entities can be vectorized using a BERT (Bidirectional ncoder Representations from Transformers) model, for example, by inputting the entities into the BERT model to generate corresponding entity vectors. The semantic similarity between the first and second target entities can be based on the cosine similarity between the first and second entity vectors.

[0052] Normalize the co-occurrence frequencies of the first and second target entities. Specifically, based on the maximum co-occurrence frequency, the co-occurrence frequencies are mapped to a specific range, such as [-1, 1] or [0, 1]. This standardizes the co-occurrence frequencies. Within the range [-1, 1], negative values indicate semantic irrelevance. In practical applications, only positive values can be retained. The maximum co-occurrence frequency is the maximum value of the co-occurrence frequency. It can be the maximum value among the historical co-occurrence frequency values in the target domain or a configured value.

[0053] In one embodiment, a GNN (Graph Neural Network) model is used to calculate the association strength between entities. The GNN model is an artificial intelligence model used to process graph-structured data. It aggregates node and edge features through a message passing mechanism to achieve tasks such as node classification and graph prediction. Based on the GNN model, the association strength between entities can be expressed as: ; (3) in, and It is the balance coefficient, also known as the weight coefficient, which is configurable and can be set based on experience. Representing an entity With entity For example, the semantic similarity between “network failure” and “slow network speed” is 0.85.

[0054] Representing an entity With entity The co-occurrence frequency between represents the maximum co-occurrence frequency, and The ratio of , which represents the normalized co-occurrence frequency, for example, the entity With entity If the entity co-occurs 50 times in the historical conversation and the maximum number of co-occurrences of entities in the target domain is 100 times, then the entity With entity The normalized co-occurrence frequency is 0.5.

[0055] When the big model generates response data, association strength is used as a decision-making basis for determining the inference path. The big model searches for nodes in the knowledge graph to implement knowledge reasoning. During the reasoning process, it prioritizes generating answers based on entity nodes with a high association strength with the current entity node. For example, if a user asks about "5G package prices," the big model can extract the entities "5G package" and "price." Among all entity nodes associated with the "5G package" entity node, the big model will prioritize the entity node with a high association strength, "Price Description," to generate the response data for the user's question.

[0056] In some embodiments, the connections between entities with low association strength in the knowledge graph can be regarded as noise, such as the mistakenly extracted entity combination "mobile phone card" and "weather query", which can be removed to filter out the noise and ensure the efficiency of the knowledge graph. Similarly, the connections between entities with high association strength can be marked as core knowledge, such as "fault code A12" and "solution", "network failure" and "restart router", etc., with a co-occurrence count of 100 times. If it exceeds the preset co-occurrence threshold, it is a high-frequency co-occurrence entity combination, and this entity combination is retained first, where the co-occurrence threshold is, for example, 50 times. If the co-occurrence count exceeds 50 times, it is a high-frequency co-occurrence entity combination.

[0057] Based on the strength of the association between entities, with the accumulation of business data in the domain dictionary and the continuous updating of the knowledge graph, the number of co-occurrences between entities continues to increase. When cross-domain business scenarios appear, based on the increase in the co-occurrence frequency between entities, the strength of the association between entities is enhanced, so that the domain knowledge in the knowledge graph can adapt to the new business scenarios and realize the knowledge transfer of domain knowledge across scenarios without the need to reconstruct the knowledge graph.

[0058] Optionally, in the initial stage, when there is a lack of co-occurrence between entities in the business data in the domain dictionary, the initial knowledge graph is constructed based on semantic similarity. In the subsequent process, as the business data in the domain dictionary continues to accumulate, the knowledge graph is continuously and dynamically optimized.

[0059] The strength of associations between entities is the glue that binds knowledge graph construction and dynamic optimization. By integrating semantic similarity with co-occurrence counts from business statistics, we calculate the weights of entity relationships and use them as the basis for decision-making in large-scale model knowledge reasoning, enabling precise representation and real-time optimization of the knowledge network. The knowledge graph's dynamic optimization mechanism enables it to adapt to business changes, filter out noise, and provide reliable decision-making basis for the large-scale model's reasoning engine, thereby enhancing the large-scale model's intelligent response to user questions.

[0060] In one embodiment, the reasoning path of the large model is optimized based on the response data generated by the large model in response to the conversation data of the user's question. After step 300, the following steps may also be included: Step 401: Obtain the indicator value of the reward indicator corresponding to the response data of the large model; the reward indicator includes response time, answer accuracy and user score; the response data is generated when the large model responds to the dialogue data; Step 402: Calculate the reward value of the reasoning path corresponding to the response data based on the indicator value and the reward function; the reasoning path is the path for performing knowledge retrieval on the domain knowledge graph according to the strength of association between domain entities in the domain knowledge graph when the large model generates the response data; the reward function is constructed based on the reward indicator; Step 403: Optimize the reasoning path according to the reward value.

[0061] After the big model generates responses to user questions, or after the user conversation ends, the reward metrics and values corresponding to the big model's responses are obtained. These reward metrics include response time, answer accuracy, and user rating. Response time refers to the time it takes the big model to retrieve domain knowledge from the knowledge graph and generate responses. Answer accuracy and user ratings can be derived based on user feedback.

[0062] Based on the obtained reward indicator values and the reward function constructed based on the reward indicators, the reward value of the reasoning path corresponding to the response data is calculated. The reward function is constructed based on the reward indicators. The reward function for the reasoning path is calculated based on the obtained reward indicator values for each reward indicator according to the calculation method represented by the reward function. The reasoning path corresponding to the response data is the logical chain formed by the large model when generating the response data. This process involves knowledge retrieval based on the entity relationships in the knowledge graph and the semantic information of the user's question, and the dynamic selection and combination of relevant knowledge nodes. The formation of this logical chain is the knowledge reasoning process of the large model.

[0063] Based on the calculated reward value, the reasoning path of the large model is optimized. The optimization methods include but are not limited to adjusting the association strength between the entity nodes included in the reasoning path, and adjusting the domain weight of each entity node included in the reasoning path.

[0064] In one embodiment, the reward function is constructed based on reinforcement learning, as shown in the following formula 4: ; (4) In formula 4, Indicates the answer accuracy of the large model, Indicates the response time of the large model, Indicates user rating. In some embodiments, response time can be and user ratings Normalization processing. 、 and Is the weight coefficient of the reward indicator, which represents the priority of the reward indicator and can be determined through A / B testing. For example, through user surveys, it is determined that the accuracy of the answer has the greatest impact on user satisfaction (accounting for 50%), followed by the response time (accounting for 30%), and the user's subjective score is used as a supplement (accounting for 20%). Then, according to the proportion of the impact of user satisfaction, the value of the weight coefficient is set. The specific setting is is 0.5, is 0.3, In actual applications, the weight coefficient can also be configured according to business needs. For example, if response speed is given priority during business peak hours, the value can be increased. The value of and The value of .

[0065] In the reinforcement learning framework, the optimization process of the reasoning path includes state parsing, action execution, reward calculation and path optimization. For state parsing, it is to parse the dialogue data of the user's questions and obtain the entity domain weights and association strengths of the knowledge graph; action execution refers to selecting an inference path from multiple candidate paths to generate response data; reward calculation is to calculate the reward value of the reasoning path based on the winning effect of the generated response data; path optimization is to optimize the reasoning path with the goal of maximizing the reward value.

[0066] In the response data generation phase, the big model can generate multiple reasoning paths based on the search results of the knowledge graph, and then select a reasoning path to generate a response based on the decision basis. For example, the conversation data of the user's question is "What does the 5G package price include?" The big model extracts the entities "5G package" and "price" from it, searches the knowledge graph based on the extracted entities, and retrieves nodes associated with the "5G package" entity from the knowledge graph, such as "price description" and "traffic quota". Based on this, multiple candidate reasoning paths can be generated, including path A: package → price description, and path B: package → traffic quota → price discount.

[0067] Furthermore, reinforcement learning techniques are used to evaluate the potential reward value of each candidate reasoning path. For example, based on the historical values of reward metrics corresponding to the candidate reasoning paths (including response time and answer accuracy), the reasoning path with the highest potential reward value is selected as the optimal path. Response data is then generated based on the domain knowledge associated with the entity nodes contained in this optimal path. This ensures that the answer not only relies on the generative capabilities of the large model but also closely integrates the logical connections of the structured knowledge graph, improving the accuracy and interpretability of the large model's responses.

[0068] In the inference path optimization stage, algorithms such as policy gradient are used to maximize the long-term cumulative reward value. In the inference path exploration stage, different inference paths are randomly adopted for training, and the reward value of each inference path is collected. In the application stage, the inference path with the highest potential reward value is selected from the generated candidate inference paths to generate response data. Finally, based on user feedback on the response data, the inference path is adjusted and optimized to adapt to business changes.

[0069] Optionally, user ratings are generated based on the user's interaction with the big model, which includes direct feedback and indirect feedback. Direct feedback can be the user directly scoring the response data, and indirect feedback is obtained based on business data, which can be conversation data obtained by users repeatedly asking the same or similar questions. The repeated questioning interaction behavior suggests that the response data is inaccurate, and the indicator value of the answer accuracy needs to be adjusted accordingly.

[0070] In this embodiment, the large model is guided to extract entities and entity relationships from unstructured conversation data. The extracted entities are then screened based on the domain dictionary to provide a structured data foundation for the construction and updating of the knowledge graph. Furthermore, by calculating the strength of associations between entities, a dynamically associated knowledge graph is constructed, resulting in a structured knowledge network that serves as the decision-making basis for the large model's inference engine. Through reinforcement learning, the large model's inference path is optimized.

[0071] Furthermore, through the closed-loop process of entity extraction-knowledge graph update-inference path optimization, the management efficiency of domain knowledge is improved, and the accuracy and real-time performance of the large model's answers are enhanced. Specifically, through the forward process of entity extraction-knowledge graph update-knowledge reasoning-response generation, and the reverse feedback process of user interaction behavior and business data feedback-inference path optimization-knowledge graph dynamic adjustment, on the one hand, the knowledge graph can be continuously updated based on real-time extracted entities, avoiding the lag of static knowledge bases. On the other hand, the static reasoning process cannot adapt to dynamically changing user needs. Based on the feedback of the large model's response data, the reasoning path is optimized, a dynamic reasoning mechanism is constructed, and the response accuracy of the large model in complex scenarios is improved.

[0072] A knowledge update mechanism based on real-time closed-loop feedback adjusts the knowledge graph and reasoning paths in real time through user interaction and business data, ensuring dynamic synchronization of domain knowledge with business needs. Furthermore, through the automated extraction of domain entities and the dynamic calculation of the strength of associations between entities, efficient integration and real-time updating of domain knowledge are achieved, shortening the knowledge update cycle.

[0073] The following describes a domain knowledge extraction device provided in an embodiment of the present application. The domain knowledge extraction device described below and the domain knowledge extraction method described above can refer to each other.

[0074] Reference Figure 2 , the domain knowledge extraction device provided in the embodiment of the present application includes: An entity extraction module 10 is configured to obtain domain entities and entity relationships in a target domain. The domain entities and entity relationships are extracted from user question conversation data by a large model based on prompt words, and the target domain is determined based on the conversation data. A confidence calculation module 20 is configured to calculate the confidence of each entity in the domain based on a domain dictionary of the target domain; the domain dictionary is accumulated based on historical conversation data of the large model in the target domain; An entity screening module 30 is used to screen the domain entities according to the confidence level to obtain target entities with high confidence level; The domain knowledge updating module 40 is used to update the knowledge graph of the target domain based on the target entity and the entity relationship corresponding to the target entity.

[0075] In one embodiment, the confidence calculation module 20 is further configured to: Based on the domain dictionary of the target domain, counting the word frequency and inverse document rate of the domain entity in the domain dictionary; Determining a domain weight of the domain entity in the target domain according to the word frequency and the inverse document rate; Obtaining entity extraction accuracy of the target domain; the entity extraction accuracy is determined based on a ratio of the number of entities in the target domain that are verified to be correctly extracted to the total number of entities extracted; The confidence of the domain entity is determined according to the entity extraction accuracy and the domain weight.

[0076] In one embodiment, the confidence calculation module 20 is further configured to: Based on the domain dictionary of the target domain, counting the frequency of occurrence of the domain entity in the domain dictionary; Determining the word frequency of the domain entity in the domain dictionary according to the occurrence frequency; Acquire a first document quantity of domain documents included in the domain dictionary, and a second document quantity of target documents including the domain entity in the domain documents; An inverse document rate of the domain entity in the domain dictionary is determined according to a ratio of the second document quantity to the first document quantity.

[0077] In one embodiment, the domain knowledge updating module 40 is further configured to: Calculating the association strength between a first target entity and each second target entity; the first target entity is any one of the target entities, and the second target entity is any one of the domain entities of the target entity and the knowledge graph of the target domain, except the first target entity; The target entity is used as a node, the entity relationship corresponding to the target entity is used as an edge, the association strength is used as the edge weight of the edge, and the knowledge graph of the target domain is updated.

[0078] In one embodiment, the domain knowledge updating module 40 is further configured to: Counting the number of co-occurrences of the first target entity and each second target entity in the target domain to obtain a co-occurrence frequency of the first target entity and the second target entity; Normalizing the co-occurrence frequency based on the maximum co-occurrence frequency corresponding to the target domain to obtain a normalized co-occurrence frequency; Vectorizing the first target entity and the second target entity to obtain a first entity vector of the first target entity and a second entity vector of the second target entity; Calculating the semantic similarity between the first target entity and the second target entity based on the first entity vector and the second entity vector; A weighted sum is performed on the normalized co-occurrence frequency and the semantic similarity to obtain the association strength between the first target entity and the second target entity.

[0079] In one embodiment, the domain knowledge extraction device further includes a reasoning path optimization module for: Obtaining the indicator value of the reward indicator corresponding to the response data of the large model; the reward indicator includes response time, answer accuracy and user score; the response data is generated when the large model responds to the dialogue data; Based on the indicator value and the reward function, a reward value of the reasoning path corresponding to the response data is calculated; the reasoning path is a path for performing knowledge retrieval on the domain knowledge graph according to the association strength between each domain entity in the domain knowledge graph when the response data is generated by the large model; the reward function is constructed based on the reward indicator; The reasoning path is optimized according to the reward value.

[0080] Figure 3 An example of a physical structure diagram of an electronic device is shown below. Figure 3As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call logic instructions in the memory 330 to execute a domain knowledge extraction method, which includes: Obtaining domain entities and entity relationships of a target domain; the domain entities and entity relationships are extracted by a large model based on prompt words from conversation data of user questions, and the target domain is determined based on the conversation data; Calculating the confidence of each entity in the domain based on a domain dictionary of the target domain; the domain dictionary is accumulated based on historical conversation data of the large model in the target domain; Screening the domain entities according to the confidence levels to obtain target entities with high confidence levels; Based on the target entity and the entity relationship corresponding to the target entity, the knowledge graph of the target domain is updated.

[0081] In addition, the logical instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program code.

[0082] On the other hand, embodiments of the present application further provide a computer program product, comprising a computer program. The computer program may be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the domain knowledge extraction method provided by each of the above methods, which includes: Obtaining domain entities and entity relationships of a target domain; the domain entities and entity relationships are extracted by a large model based on prompt words from conversation data of user questions, and the target domain is determined based on the conversation data; Calculating the confidence of each entity in the domain based on a domain dictionary of the target domain; the domain dictionary is accumulated based on historical conversation data of the large model in the target domain; Screening the domain entities according to the confidence levels to obtain target entities with high confidence levels; Based on the target entity and the entity relationship corresponding to the target entity, the knowledge graph of the target domain is updated.

[0083] In another aspect, embodiments of the present application further provide a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for extracting domain knowledge provided by the above methods is implemented, and the method includes: Obtaining domain entities and entity relationships of a target domain; the domain entities and entity relationships are extracted by a large model based on prompt words from conversation data of user questions, and the target domain is determined based on the conversation data; Calculating the confidence of each entity in the domain based on a domain dictionary of the target domain; the domain dictionary is accumulated based on historical conversation data of the large model in the target domain; Screening the domain entities according to the confidence levels to obtain target entities with high confidence levels; Based on the target entity and the entity relationship corresponding to the target entity, the knowledge graph of the target domain is updated.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0085] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for extracting domain knowledge, characterized in that: include: Get the domain entities and entity relationships of the target domain; The domain entities and the entity relationships are extracted by a large model based on prompt words from conversation data of user questions, and the target domain is determined based on the conversation data; Calculating the confidence of each entity in the domain based on a domain dictionary of the target domain; the domain dictionary is accumulated based on historical conversation data of the large model in the target domain; Screening the domain entities according to the confidence levels to obtain target entities with high confidence levels; Based on the target entity and the entity relationship corresponding to the target entity, the knowledge graph of the target domain is updated.

2. The domain knowledge extraction method according to claim 1, characterized in that: The calculating of the confidence of each domain entity based on the domain dictionary of the target domain includes: Based on the domain dictionary of the target domain, counting the word frequency and inverse document rate of the domain entity in the domain dictionary; Determining a domain weight of the domain entity in the target domain according to the word frequency and the inverse document rate; Obtaining entity extraction accuracy of the target domain; the entity extraction accuracy is determined based on a ratio of the number of entities in the target domain that are verified to be correctly extracted to the total number of entities extracted; The confidence of the domain entity is determined according to the entity extraction accuracy and the domain weight.

3. The domain knowledge extraction method according to claim 2, characterized in that: The counting of the word frequency and inverse document rate of the domain entity in the domain dictionary based on the target domain includes: Based on the domain dictionary of the target domain, counting the frequency of occurrence of the domain entity in the domain dictionary; Determining the word frequency of the domain entity in the domain dictionary according to the occurrence frequency; Acquire a first document quantity of domain documents included in the domain dictionary, and a second document quantity of target documents including the domain entity in the domain documents; An inverse document rate of the domain entity in the domain dictionary is determined according to a ratio of the second document quantity to the first document quantity.

4. The domain knowledge extraction method according to claim 1, characterized in that: The updating of the knowledge graph of the target domain based on the target entity and the entity relationship corresponding to the target entity includes: Calculating the association strength between a first target entity and each second target entity; the first target entity is any one of the target entities, and the second target entity is any one of the domain entities of the target entity and the knowledge graph of the target domain, except the first target entity; The target entity is used as a node, the entity relationship corresponding to the target entity is used as an edge, the association strength is used as the edge weight of the edge, and the knowledge graph of the target domain is updated.

5. The domain knowledge extraction method according to claim 4, characterized in that: The calculating the association strength between the first target entity and the second target entity includes: Counting the number of co-occurrences of the first target entity and each second target entity in the target domain to obtain a co-occurrence frequency of the first target entity and the second target entity; Normalizing the co-occurrence frequency based on the maximum co-occurrence frequency corresponding to the target domain to obtain a normalized co-occurrence frequency; Vectorizing the first target entity and the second target entity to obtain a first entity vector of the first target entity and a second entity vector of the second target entity; Calculating the semantic similarity between the first target entity and the second target entity based on the first entity vector and the second entity vector; A weighted sum is performed on the normalized co-occurrence frequency and the semantic similarity to obtain the association strength between the first target entity and the second target entity.

6. The domain knowledge extraction method according to claim 4, characterized in that: After updating the knowledge graph of the target domain based on the target entity and the entity relationship corresponding to the target entity, the method further includes: Obtaining the indicator value of the reward indicator corresponding to the response data of the large model; the reward indicator includes response time, answer accuracy and user score; the response data is generated when the large model responds to the dialogue data; Based on the indicator value and the reward function, a reward value of the reasoning path corresponding to the response data is calculated; the reasoning path is a path for performing knowledge retrieval on the domain knowledge graph according to the association strength between each domain entity in the domain knowledge graph when the response data is generated by the large model; the reward function is constructed based on the reward indicator; The reasoning path is optimized according to the reward value.

7. A domain knowledge extraction device, characterized in that: include: Entity extraction module, used to obtain domain entities and entity relationships in the target domain; The domain entities and the entity relationships are extracted by a large model based on prompt words from conversation data of user questions, and the target domain is determined based on the conversation data; A confidence calculation module, configured to calculate the confidence of each entity in the domain based on a domain dictionary of the target domain; the domain dictionary is accumulated based on historical conversation data of the large model in the target domain; An entity screening module, configured to screen the domain entities according to the confidence level to obtain target entities with high confidence levels; A domain knowledge updating module is used to update the knowledge graph of the target domain based on the target entity and the entity relationship corresponding to the target entity.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the domain knowledge extraction method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the domain knowledge extraction method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the domain knowledge extraction method according to any one of claims 1 to 6 is implemented.

Images